```{raw} html

```

*optimagic* is a Python package for numerical optimization. It is a unified interface to optimizers from SciPy, NlOpt and many other Python packages. *optimagic*'s `minimize` function works just like SciPy's, so you don't have to adjust your code. You simply get more optimizers for free. On top you get powerful diagnostic tools, parallel numerical derivatives and more. If you want to see what *optimagic* can do, check out this [tutorial](tutorials/optimization_overview.ipynb) *optimagic* was formerly called *estimagic*, because it also provides functionality to perform statistical inference on estimated parameters. *estimagic* is now a subpackage of *optimagic*, which is documented [here](estimagic). `````{grid} 1 2 2 2 --- gutter: 3 --- ````{grid-item-card} :text-align: center :img-top: _static/images/light-bulb.svg :class-img-top: index-card-image :shadow: md ```{button-link} tutorials/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Tutorials ``` New users of optimagic should read this first. ```` ````{grid-item-card} :text-align: center :img-top: _static/images/book.svg :class-img-top: index-card-image :shadow: md ```{button-link} how_to/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- How-to Guides ``` Detailed instructions for specific and advanced tasks. ```` ````{grid-item-card} :text-align: center :img-top: _static/images/installation.svg :class-img-top: index-card-image :shadow: md ```{button-link} installation.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Installation ``` Installation instructions for optimagic and optional dependencies. ```` ````{grid-item-card} :text-align: center :img-top: _static/images/optimization.svg :class-img-top: index-card-image :shadow: md ```{button-link} algorithms.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Optimization Algorithms ``` List of numerical optimizers and their optional parameters. ```` ````{grid-item-card} :text-align: center :img-top: _static/images/books.svg :class-img-top: index-card-image :shadow: md ```{button-link} explanation/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Explanations ``` Background information on key topics central to the package. ```` ````{grid-item-card} :text-align: center :img-top: _static/images/coding.svg :class-img-top: index-card-image :shadow: md ```{button-link} reference/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- API Reference ``` Detailed description of the optimagic API. ```` ````{grid-item-card} :text-align: center :columns: 12 :img-top: _static/images/video.svg :class-img-top: index-card-image :shadow: md ```{button-link} videos.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Videos ``` Collection of tutorials, talks, and screencasts on optimagic. ```` ````` ```{toctree} --- hidden: true maxdepth: 1 --- tutorials/index how_to/index explanation/index reference/index development/index videos algorithms estimagic/index installation ``` ______________________________________________________________________ We thank all institutions that have funded or supported optimagic (formerly estimagic) ```{image} _static/images/aai-institute-logo.svg --- width: 185px --- ``` ```{image} _static/images/numfocus_logo.png --- width: 200 --- ``` ```{image} _static/images/tra_logo.png --- width: 240px --- ``` ```{image} _static/images/hoover_logo.png --- width: 192px --- ``` ```{image} _static/images/transferlab-logo.svg --- width: 420px --- ``` ______________________________________________________________________ **Useful links for search:** {ref}`genindex` | {ref}`modindex` | {ref}`search` (tutorials)= # Tutorials This section provides an overview of optimagic. It's a good starting point if you are new to optimagic. For more in-depth examples using advanced options, check out the [how-to guides](how-to). `````{grid} 1 2 2 3 --- gutter: 3 --- ````{grid-item-card} :text-align: center :img-top: ../_static/images/optimization.svg :class-img-top: index-card-image :shadow: md ```{button-link} optimization_overview.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Optimization ``` Learn numerical optimization with estimagic. ```` ````{grid-item-card} :text-align: center :img-top: ../_static/images/differentiation.svg :class-img-top: index-card-image :shadow: md ```{button-link} numdiff_overview.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Differentiation ``` Learn numerical differentiation with estimagic. ```` ````{grid-item-card} :text-align: center :img-top: ../_static/images/bayesian_optimization.svg :class-img-top: index-card-image :shadow: md ```{button-link} bayes_opt_tutorial.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- bayes_opt Optimizer ``` Tutorial on the bayes_opt optimizer in optimagic. ```` ````` ```{toctree} --- hidden: true maxdepth: 1 --- optimization_overview numdiff_overview bayes_opt_tutorial ``` { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Numerical optimization\n", "\n", "Using simple examples, this tutorial shows how to do an optimization with optimagic. More details on the topics covered here can be found in the [how to guides](../how_to/index.md)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic usage of `minimize`\n", "\n", "The basic usage of `optimagic.minimize` is very similar to `scipy.optimize.minimize`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere(params):\n", " return params @ params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "lbfgsb_res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "\n", "lbfgsb_res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## `params` do not have to be vectors\n", "\n", "In optimagic, params can by arbitrary [pytrees](https://jax.readthedocs.io/en/latest/pytrees.html). Examples are (nested) dictionaries of numbers, arrays and pandas objects. This is very useful if you have many parameters!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def dict_sphere(params):\n", " return params[\"a\"] ** 2 + params[\"b\"] ** 2 + (params[\"c\"] ** 2).sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "nm_res = om.minimize(\n", " fun=dict_sphere,\n", " params={\"a\": 0, \"b\": 1, \"c\": pd.Series([2, 3, 4])},\n", " algorithm=\"scipy_neldermead\",\n", ")\n", "\n", "nm_res.params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## You can compare optimizers\n", "\n", "In practice, it is super hard to pick the right optimizer for your problem. With optimagic, you can simply try a few and compare their results!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results = {\"lbfgsb\": lbfgsb_res, \"nelder_mead\": nm_res}\n", "fig = om.criterion_plot(results, max_evaluations=300)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "\n", "For details on using other plotting backends, see [How to change the plotting backend](../how_to/how_to_change_plotting_backend.ipynb).\n", "\n", ":::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can also zoom in on the history of specific parameters. This can be super helpful to diagnose problems in the optimization. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.params_plot(\n", " nm_res,\n", " max_evaluations=300,\n", " # optionally select a subset of parameters to plot\n", " selector=lambda params: params[\"c\"],\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## There are many optimizers\n", "\n", "By default, optimagic comes with optimizers from scipy, including global optimizers \n", "and least-squares optimizers. But we also have wrappers for algorithms from **NlOpt**, \n", "**Pygmo**, as well as several optimizers from individual packages like **fides**, \n", "**ipopt**, **pybobyqa** and **dfols**. \n", "\n", "To use optimizers that are not from scipy, follow our [installation guide](../installation.md) for optional dependencies. To see which optimizers we have, check out the [full list](../algorithms.md).\n", "\n", "If you are missing your favorite optimizer in the list, let us know with an [issue](https://github.com/optimagic-dev/optimagic/issues)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Amazing autocomplete \n", "\n", "Assume you need a gradient-free optimizer that supports bounds on the parameters. Moreover, you have a fixed computational budget, so you want to set stopping options. \n", "\n", "In most optimizer libraries, you would have to spend a few minutes with the docs to find an optimizer that fits your needs and the stopping options it supports. In optimagic, all of this is discoverable in your editor!\n", "\n", "If you type `om.algos.`, your editor will show you all available optimizers and a list of categories you can use to filter the results. In our case, we select `GradientFree` and `Bounded`, and we could do that in any order we want.\n", "\n", "\n", "![autocomplete_1](../_static/images/autocomplete_1.png)\n", "\n", "\n", "After selecting one of the displayed algorithms, in our case `scipy_neldermead`, the editor shows all tuning parameters of that optimizer. If you start to type `stopping`, you will see all stopping criteria that are available.\n", "\n", "\n", "![autocomplete_2](../_static/images/autocomplete_2.png)\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding bounds\n", "\n", "As any optimizer library, optimagic lets you specify bounds for the parameters." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bounds = om.Bounds(lower=np.arange(5) - 2, upper=np.array([10, 10, 10, np.inf, np.inf]))\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=bounds,\n", ")\n", "\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fixing parameters \n", "\n", "On top of bounds, you can also fix one or more parameters during the optimization. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " constraints=om.FixedConstraint(selector=lambda params: params[[1, 3]]),\n", ")\n", "\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Other constraints\n", "\n", "As an example, let's impose the constraint that the first three parameters are valid probabilities, i.e. they are between zero and one and sum to one:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere,\n", " params=np.array([0.1, 0.5, 0.4, 4, 5]),\n", " algorithm=\"scipy_lbfgsb\",\n", " constraints=om.ProbabilityConstraint(selector=lambda params: params[:3]),\n", ")\n", "\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For a full overview of the constraints we support and the corresponding syntaxes, check out [the documentation](../how_to/how_to_constraints.md).\n", "\n", "Note that `\"scipy_lbfgsb\"` is not a constrained optimizer. If you want to know how we achieve this, check out [the explanations](../explanation/implementation_of_constraints.md)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## There is also maximize\n", "\n", "If you ever forgot to switch back the sign of your criterion function after doing a maximization with `scipy.optimize.minimize`, there is good news:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def upside_down_sphere(params):\n", " return -params @ params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.maximize(\n", " fun=upside_down_sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_bfgs\",\n", ")\n", "\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "optimagic got your back." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Speeding up your optimization with derivatives \n", "\n", "You can speed up your optimization by providing closed form derivatives. Those derivatives can be hand-coded or calculated with JAX!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere_gradient(params):\n", " return 2 * params\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " jac=sphere_gradient,\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, you can let optimagic calculate numerical derivatives with parallelized finite differences. This is very handy if you do not want to invest the time to derive the derivatives of your criterion function. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " numdiff_options=om.NumdiffOptions(n_cores=6),\n", ")\n", "\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more details and examples check out [how-to speed up your optimization with derivatives](../how_to/how_to_derivatives.ipynb)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Turn local optimizers global with multistart\n", "\n", "Multistart optimization requires finite soft bounds on all parameters. Those bounds will\n", "be used for sampling but not enforced during optimization." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "bounds = om.Bounds(soft_lower=np.full(10, -5), soft_upper=np.full(10, 15))\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(10),\n", " algorithm=\"scipy_neldermead\",\n", " bounds=bounds,\n", " multistart=om.MultistartOptions(convergence_max_discoveries=5),\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## And plot the history of all local optimizations" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(res)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exploit the structure of your optimization problem\n", "\n", "Many estimation problems have a least-squares structure. If so, specialized optimizers that exploit this structure can be much faster than standard optimizers. The `sphere` function from above is the simplest possible least-squarse problem you could imagine: the least-squares residuals are just the params. \n", "\n", "To use least-squares optimizers in optimagic, you need to declare mark your function with \n", "a decorator and return the least-squares residuals instead of the aggregated function value. \n", "\n", "More details can be found [here](../how_to/how_to_criterion_function.md)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "@om.mark.least_squares\n", "def ls_sphere(params):\n", " return params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=ls_sphere,\n", " params=np.arange(5),\n", " algorithm=\"pounders\",\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, any least-squares problem can also be solved with a standard optimizer. \n", "\n", "There are also specialized optimizers for likelihood functions. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using and reading persistent logging\n", "\n", "For long-running and difficult optimizations, it can be worthwhile to store the progress in a persistent log file. You can do this by providing a path to the `logging` argument:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " logging=\"my_log.db\",\n", " log_options={\"if_database_exists\": \"replace\"},\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can read the entries in the log file (while the optimization is still running or after it has finished) as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader = om.OptimizeLogReader(\"my_log.db\")\n", "reader.read_history().keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For more information on what you can do with the log file and LogReader object, check out [the logging tutorial](../how_to/how_to_logging.ipynb)\n", "\n", "The persistent log file is always instantly synchronized when the optimizer tries a new parameter vector. This is very handy if an optimization has to be aborted and you want to extract the current status. It can be displayed in `criterion_plot` and `params_plot`, even while the optimization is running. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" }, "vscode": { "interpreter": { "hash": "40d3a090f54c6569ab1632332b64b2c03c39dcf918b08424e98f38b5ae0af88f" } } }, "nbformat": 4, "nbformat_minor": 4 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Numerical differentiation\n", "\n", "In this tutorial, you will learn how to numerically differentiate functions with\n", "optimagic." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "\n", "import optimagic as om" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic usage of `first_derivative`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def fun(params):\n", " return params @ params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fd = om.first_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", ")\n", "\n", "fd.derivative" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Basic usage of `second_derivative`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sd = om.second_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", ")\n", "\n", "sd.derivative.round(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## You can parallelize" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fd = om.first_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", " n_cores=4,\n", ")\n", "\n", "fd.derivative" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sd = om.second_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", " n_cores=4,\n", ")\n", "\n", "sd.derivative.round(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## `params` do not have to be vectors\n", "\n", "In optimagic, params can be arbitrary [pytrees](https://jax.readthedocs.io/en/latest/pytrees.html). Examples are (nested) dictionaries of numbers, arrays and pandas objects. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def dict_fun(params):\n", " return params[\"a\"] ** 2 + params[\"b\"] ** 2 + (params[\"c\"] ** 2).sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fd = om.first_derivative(\n", " func=dict_fun,\n", " params={\"a\": 0, \"b\": 1, \"c\": pd.Series([2, 3, 4])},\n", ")\n", "\n", "fd.derivative" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Description of the output\n", "\n", "> Note. Understanding the output of the first and second derivative requires terminolgy\n", "> of pytrees. Please refer to the\n", "> [JAX documentation of pytrees](https://jax.readthedocs.io/en/latest/pytrees.html).\n", "\n", "The output tree of `first_derivative` has the same structure as the params tree.\n", "Equivalent to the 1-d numpy array case, where the gradient is a vector of shape\n", "`(len(params),)`. If, however, the params tree contains non-scalar entries like\n", "`numpy.ndarray`'s, `pandas.Series`', or `pandas.DataFrame`'s, the output is not expanded\n", "but a block is created instead. In the above example, the entry `params[\"c\"]` is a\n", "`pandas.Series` with 3 entries. Thus, the first derivative output contains the\n", "corresponding 3x1-block of the gradient at the position `[\"c\"]`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fd.derivative[\"c\"]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sd = om.second_derivative(\n", " func=dict_fun,\n", " params={\"a\": 0, \"b\": 1, \"c\": pd.Series([2, 3, 4])},\n", ")\n", "\n", "sd.derivative" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Description of the output\n", "\n", "> Note. Understanding the output of the first and second derivative requires terminolgy\n", "> of pytrees. Please refer to the\n", "> [JAX documentation of pytrees](https://jax.readthedocs.io/en/latest/pytrees.html).\n", "\n", "The output of `second_derivative` when using a general pytrees looks more complex but\n", "is easy once we remember that the second derivative is equivalent to applying the first\n", "derivative twice.\n", "\n", "The output tree is a product of the params tree with itself. This is equivalent to the\n", "1-d numpy array case, where the hessian is a matrix of shape\n", "`(len(params), len(params))`. If, however, the params tree contains non-scalar entries\n", "like `numpy.ndarray`'s, `pandas.Series`', or `pandas.DataFrame`'s, the output is not\n", "expanded but a block is created instead. In the above example, the entry `params[\"c\"]`\n", "is a 3-dimensional `pandas.Series`. Thus, the second derivative output contains the\n", "corresponding 3x3-block of the hessian at the position `[\"c\"][\"c\"]`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sd.derivative[\"c\"][\"c\"].round(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## There are many options\n", "\n", "You can choose which finite difference method to use, whether we should respect\n", "parameter bounds, or whether to evaluate the function in parallel. Let's go through\n", "some basic examples. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## You can choose the difference method\n", "\n", "> Note. A mathematical explanation of the background of the difference methods can be\n", "> found on the corresponding [explanation page](../explanation/numdiff_background.md)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fd = om.first_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", " method=\"backward\", # default: 'central'\n", ")\n", "\n", "fd.derivative" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sd = om.second_derivative(\n", " func=fun,\n", " params=np.arange(5),\n", " method=\"forward\", # default: 'central_cross'\n", ")\n", "\n", "sd.derivative.round(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## You can add bounds " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "params = np.arange(5)\n", "\n", "fd = om.first_derivative(\n", " func=fun,\n", " params=params,\n", " # forces first_derivative to use forward differences\n", " bounds=om.Bounds(lower=params, upper=params + 1),\n", ")\n", "\n", "fd.derivative" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, bounds also work in `second_derivative`." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" }, "vscode": { "interpreter": { "hash": "40d3a090f54c6569ab1632332b64b2c03c39dcf918b08424e98f38b5ae0af88f" } } }, "nbformat": 4, "nbformat_minor": 4 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# `bayes_opt` Optimizer in optimagic" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "This tutorial demonstrates how to use the `\"bayes_opt\"` optimizer in optimagic. To use it, you need to have `bayesian-optimization` package installed. You can install it with the following command:\n", "```bash\n", "pip install bayesian-optimization\n", "```" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "### When to use Bayesian Optimization:\n", "- Function evaluations are expensive (e.g., simulations, experiments)\n", "- The function is a black box(it cannot be expressed in closed form)\n", "- You have a limited budget of function evaluations\n", "- When gradients are unavailable or computationally expensive to obtain\n", "\n", "### Key Concepts\n", "\n", "### Gaussian Processes (GP)\n", "The GP serves as a probabilistic model of your objective function. It provides both a mean prediction and uncertainty estimates.\n", "### Acquisition Functions\n", "These functions use the GP's predictions to decide where to evaluate next.\n", "\n", "Common acquisition functions include:\n", "- **Upper Confidence Bound (UCB)**: Balances mean prediction with uncertainty\n", "- **Expected Improvement (EI)**: Expected improvement over the current best\n", "- **Probability of Improvement (POI)**: Probability of improving over the current best" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "\n", "import optimagic as om\n", "from bayes_opt import acquisition" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## Basic Usage of the `bayes_opt` Optimizer" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "Let's start with a simple example using a sphere function" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "def sphere(params):\n", " return params @ params" ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# Note: bayes_opt requires bounds for all parameters\n", "bounds = om.Bounds(\n", " lower=np.full(2, -10.0),\n", " upper=np.full(2, 10.0)\n", ")\n", "bayesopt_res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(2),\n", " algorithm=\"bayes_opt\",\n", " bounds=bounds,\n", " algo_options={\"seed\": 1}\n", ")\n", "\n", "bayesopt_res.params" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "## Acquisition Functions in the `bayes_opt` Optimizer" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "In Bayesian optimization, the **acquisition function** decides *where to sample next*.\n", "It controls the trade-off between **exploration** (search new areas) and **exploitation** (focus on good areas).\n", "\n", "optimagic lets you set the acquisition function in different ways:" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "### 1. Using No Acquisition Function (Default)" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# Uses package defaults: UCB for unconstrained, EI for constrained\n", "acquisition_function = None" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "### 2. Using a String (Built-in acquisition functions)\n", "\n", "You can pass any of the following strings to select a standard acquisition function:\n", "\n", "* `\"ucb\"` / `\"upper_confidence_bound\"` – Upper Confidence Bound\n", "* `\"ei\"` / `\"expected_improvement\"` – Expected Improvement\n", "* `\"poi\"` / `\"probability_of_improvement\"` – Probability of Improvement" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "acquisition_function_str = \"ucb\"" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "### 3. Using a Class (Auto-Instantiated)\n", "\n", "You can also pass the class directly, optimagic will create an instance for it:" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "from bayes_opt.acquisition import UpperConfidenceBound\n", "\n", "acquisition_function_class = UpperConfidenceBound" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "### 4. Using an Instance" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "from bayes_opt.acquisition import ExpectedImprovement\n", "\n", "acquisition_function_instance = ExpectedImprovement(\n", " xi=0.1,\n", " exploration_decay=0.95,\n", " exploration_decay_delay=5\n", ")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "### Example Run with configured acquisition functions" ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere ,\n", " params=np.arange(2),\n", " algorithm=\"bayes_opt\",\n", " bounds=om.Bounds(lower=np.full(2, -5.0), upper=np.full(2, 5.0)),\n", " algo_options={\"seed\":1, \"acquisition_function\": acquisition_function_str,}\n", " # acquisition_function can be any of:\n", " # acquisition_function_str → e.g. \"ucb\", \"ei\", \"poi\"\n", " # acquisition_function_class → e.g. UpperConfidenceBound\n", " # acquisition_function_instance → e.g. ExpectedImprovement(xi=0.1)\n", " # None → defaults to \"ucb\"\n", " )\n", "\n", "res.params" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "## Custom Acquisition Functions" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "`bayesian-optimization` also allows us to write our own acquisition functions by subclassing its `AcquisitionFunction` class. This allows you to define exploration/exploitation strategies tailored to your specific problem." ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "### Implementation Requirements\n", "\n", "When subclassing `AcquisitionFunction`, there are specific methods we must implement:\n", "\n", "1. **`base_acq(self, mean, std)` method (Required)**: This is the core method where you define the mathematical formula for your acquisition function. It takes the predicted mean and standard deviation from the Gaussian Process and returns the acquisition value(s).\n", "\n", "2. **`suggest` method (Optional but often needed)**: The base class provides a default implementation, but you may need to override it if you need to set up internal state (like `y_max` for EI/PI) before `base_acq` is called.\n", "\n", "3. **`get_acquisition_params` and `set_acquisition_params` methods (Optional but recommended)**: These are used for retrieving and setting the internal parameters of your acquisition function. Implementing them makes your acquisition function fully configurable and serializable." ] }, { "cell_type": "code", "execution_count": null, "id": "23", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "from bayes_opt.acquisition import AcquisitionFunction\n", "class CustomAcquisition(AcquisitionFunction):\n", " \"\"\"\n", " A simple custom acquisition function.\n", "\n", " This acquisition returns half of the predicted mean.\n", " It ignores the uncertainty (std), making it purely\n", " exploitation-oriented.\n", " \"\"\"\n", " def __init__(self):\n", " super().__init__()\n", "\n", " def base_acq(self, mean, std):\n", " return 0.5 * mean" ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "### Using the Custom Acquisition Function\n", "\n", "Once you have defined your custom acquisition function, you can use it in optimagic by passing an instance or a class to the `acquisition_function` parameter:" ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "acquisition_function = CustomAcquisition()\n", "\n", "res = om.minimize(\n", " fun=sphere ,\n", " params=np.arange(2),\n", " algorithm=\"bayes_opt\",\n", " bounds=om.Bounds(lower=np.full(2, -5.0), upper=np.full(2, 5.0)),\n", " algo_options={\"seed\":1, \"acquisition_function\": acquisition_function,}\n", " )\n", "\n", "res.params" ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "### Meta Acquisition Functions" ] }, { "cell_type": "markdown", "id": "27", "metadata": {}, "source": [ "The `bayesian-optimization` package also provides meta acquisition functions that operate on other acquisition functions:\n", "\n", "1. **GPHedge**: Dynamically chooses the best acquisition function from a set of candidates based on their past performance.\n", "2. **ConstantLiar**: Used for parallelized optimization to discourage sampling near points that have already been suggested but not yet evaluated.\n", "\n", "Here's how to use GPHedge with multiple base acquisition functions:" ] }, { "cell_type": "markdown", "id": "28", "metadata": {}, "source": [ "### 1. **GPHedge**:\n", "Dynamically chooses the best acquisition function from a set of candidates based on their past performance.\n", "\n", "let’s define the **Branin function**, to use with Meta Acquisition functions." ] }, { "cell_type": "code", "execution_count": null, "id": "29", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "def branin(params):\n", " \"\"\"The Branin function - a classic optimization test function.\n", "\n", " Has three global minima at approximately:\n", " - (-π, 12.275)\n", " - (π, 2.275)\n", " - (9.42478, 2.475)\n", "\n", " Global minimum value: 0.397887\n", " \"\"\"\n", " x1, x2 = params[0], params[1]\n", "\n", " a = 1\n", " b = 5.1 / (4 * np.pi**2)\n", " c = 5 / np.pi\n", " r = 6\n", " s = 10\n", " t = 1 / (8 * np.pi)\n", "\n", " term1 = a * (x2 - b * x1**2 + c * x1 - r)**2\n", " term2 = s * (1 - t) * np.cos(x1)\n", " term3 = s\n", "\n", " return term1 + term2 + term3" ] }, { "cell_type": "code", "execution_count": null, "id": "30", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "from bayes_opt.acquisition import GPHedge, UpperConfidenceBound, ExpectedImprovement\n", "\n", "# Create a list of base acquisition functions\n", "base_acquisitions = [\n", " UpperConfidenceBound(kappa=2.576),\n", " ExpectedImprovement(xi=0.01),\n", " # Add more as needed\n", "]\n", "\n", "gphedge_acq = GPHedge(base_acquisitions)\n", "\n", "result = om.minimize(\n", " fun=branin,\n", " params=np.array([1.0, 1.0]),\n", " algorithm=\"bayes_opt\",\n", " bounds=bounds,\n", " algo_options={\n", " \"acquisition_function\": gphedge_acq,\n", " \"seed\": 42\n", " }\n", ")\n", "\n", "result.params, result.fun" ] }, { "cell_type": "markdown", "id": "31", "metadata": {}, "source": [ "### 2. ConstantLiar\n", "\n", "`ConstantLiar` is used for parallelized optimization. It discourages sampling near points that have already been suggested but not yet evaluated." ] }, { "cell_type": "code", "execution_count": null, "id": "32", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "from bayes_opt.acquisition import ConstantLiar, UpperConfidenceBound\n", "\n", "base_acq = UpperConfidenceBound(kappa=2.576)\n", "\n", "constant_liar_acq = ConstantLiar(base_acquisition=base_acq, strategy=\"max\")\n", "\n", "# Use in optimization (Note: ConstantLiar is primarily for async optimization)\n", "result = om.minimize(\n", " fun=sphere,\n", " params=np.array([1.0, 1.0]),\n", " algorithm=\"bayes_opt\",\n", " bounds=bounds,\n", " algo_options={\n", " \"acquisition_function\": constant_liar_acq,\n", " \"seed\": 42\n", " }\n", ")\n", "\n", "result.params" ] }, { "cell_type": "markdown", "id": "33", "metadata": {}, "source": [ "## Exploration vs Exploitation Trade-off" ] }, { "cell_type": "markdown", "id": "34", "metadata": {}, "source": [ "When using Bayesian optimization, the acquisition function decides where to sample next. It balances exploration (try new areas) vs exploitation (refine known good areas).\n", "\n", "- **Exploration**: Sampling in regions with high uncertainty\n", "- **Exploitation**: Sampling in regions with high predicted values\n", "\n", "### Related Parameters\n", "\n", "- **kappa** (UCB): Higher values → more exploration\n", "- **xi** (EI/POI): Higher values → more exploration\n", "- **exploration_decay**: Gradually shift from exploration to exploitation\n", "- **exploration_decay_delay**: When to start the decay" ] }, { "cell_type": "code", "execution_count": null, "id": "35", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "def f(x):\n", " \"\"\"Function with multiple peaks\"\"\"\n", " x = x[0]\n", " return float(\n", " np.exp(-(x - 2) ** 2) +\n", " np.exp(-(x - 6) ** 2 / 10) +\n", " 1 / (x ** 2 + 1)\n", " )\n", "x = np.linspace(-2, 10, 100)\n", "Y = [f([xi]) for xi in x]\n", "plt.plot(x, Y)\n", "plt.xlabel(\"x\")\n", "plt.ylabel(\"f(x)\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "id": "36", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "def plot_bayes_opt(result):\n", " \"\"\"Plot optimization results\"\"\"\n", " evaluated_points = np.array([p[0] for p in result.history.params])\n", " function_values = np.array(result.history.fun)\n", "\n", " plt.figure(figsize=(8,5))\n", " plt.plot(x, Y, 'b-', label=\"Original function f(x)\")\n", " plt.scatter(evaluated_points, function_values, c=\"red\", s=60, zorder=3, label=\"Evaluated points\")\n", " plt.axvline(result.params[0], color=\"green\", linestyle=\"--\", label=\"Best param\")\n", "\n", " plt.xlabel(\"x\")\n", " plt.ylabel(\"f(x)\")\n", " plt.legend()\n", " plt.grid(True, alpha=0.3)\n", " plt.show()\n" ] }, { "cell_type": "code", "execution_count": null, "id": "37", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# strategy: exploitation (kappa=0.1) - focuses on known good areas\n", "acquisition_function = acquisition.UpperConfidenceBound(kappa=0.1)\n", "result = om.maximize(\n", " fun=f,\n", " params=np.array([0.]),\n", " algorithm=\"bayes_opt\",\n", " bounds=om.Bounds(lower=np.full(1, -2.0), upper=np.full(1, 10.0)),\n", " algo_options={\n", " \"acquisition_function\": acquisition_function,\n", " \"seed\": 987234,\n", " }\n", ")\n", "\n", "# Notice: Points cluster around peaks, might also get stuck in local optimum\n", "plot_bayes_opt(result)" ] }, { "cell_type": "code", "execution_count": null, "id": "38", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# strategy: exploration (kappa=10) - explores more broadly\n", "acquisition_function = acquisition.UpperConfidenceBound(kappa=10)\n", "result = om.maximize(\n", " fun=f,\n", " params=np.array([0.]),\n", " algorithm=\"bayes_opt\",\n", " bounds=om.Bounds(lower=np.full(1, -2.0), upper=np.full(1, 10.0)),\n", " algo_options={\n", " \"acquisition_function\": acquisition_function,\n", " \"seed\": 987234,\n", " }\n", ")\n", "\n", "# Notice: Points are more spread out, better chance of finding global optimum\n", "plot_bayes_opt(result)" ] }, { "cell_type": "markdown", "id": "39", "metadata": {}, "source": [ "## Sequential Domain Reduction (SDR)" ] }, { "cell_type": "markdown", "id": "40", "metadata": {}, "source": [ "Sequential Domain Reduction (SDR) progressively narrows the search space around promising regions. This can significantly improve optimization, especially for high-dimensional problems.\n", "\n", "### SDR Parameters\n", "\n", "- `enable_sdr`: Enable/disable Sequential Domain Reduction\n", "- `sdr_gamma_osc`: Controls oscillation damping (default: 0.7)\n", "- `sdr_gamma_pan`: Controls panning behavior (default: 1.0)\n", "- `sdr_eta`: Zooming parameter for region shrinking (default: 0.9)\n", "- `sdr_minimum_window`: Minimum window size (default: 0.0)" ] }, { "cell_type": "markdown", "id": "41", "metadata": {}, "source": [ "### SDR Example" ] }, { "cell_type": "code", "execution_count": null, "id": "42", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "def ackley(x):\n", " \"\"\"Global minimum: f(x*) = 0 at x* = (0, 0)\"\"\"\n", " x0, x1 = x\n", " arg1 = -0.2 * np.sqrt(0.5 * (x0 ** 2 + x1 ** 2))\n", " arg2 = 0.5 * (np.cos(2 * np.pi * x0) + np.cos(2 * np.pi * x1))\n", " return -20. * np.exp(arg1) - np.exp(arg2) + 20. + np.e\n", "\n", "start_params = np.array([2.0, 2.0])\n", "bounds = om.Bounds(\n", " lower=np.array([-32.768, -32.768]),\n", " upper=np.array([32.768, 32.768])\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "43", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# Standard Bayesian Optimization without SDR\n", "result_standard = om.minimize(\n", " fun=ackley,\n", " params=start_params,\n", " algorithm=\"bayes_opt\",\n", " bounds=bounds,\n", " algo_options={\n", " \"enable_sdr\": False,\n", " \"n_iter\": 50,\n", " \"init_points\": 2,\n", " \"seed\": 1,\n", " \"acquisition_function\": \"ucb\",\n", " }\n", ")\n", "\n", "print(\"Standard Bayesian Optimization:\")\n", "print(\"Best function value:\", result_standard.fun)\n", "print(\"Best parameters:\", result_standard.x)" ] }, { "cell_type": "code", "execution_count": null, "id": "44", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# Bayesian Optimization with SDR\n", "result_sdr = om.minimize(\n", " fun=ackley,\n", " params=start_params,\n", " algorithm=\"bayes_opt\",\n", " bounds=bounds,\n", " algo_options={\n", " \"enable_sdr\": True,\n", " \"sdr_minimum_window\": 0.5,\n", " \"sdr_gamma_osc\": 0.7,\n", " \"sdr_gamma_pan\": 1.0,\n", " \"sdr_eta\": 0.9,\n", " \"n_iter\": 50,\n", " \"init_points\": 2,\n", " \"seed\": 1,\n", " \"acquisition_function\": \"ucb\",\n", " }\n", ")\n", "\n", "print(\"Bayesian Optimization with SDR:\")\n", "print(\"Best function value:\", result_sdr.fun)\n", "print(\"Best parameters:\", result_sdr.x)" ] }, { "cell_type": "code", "execution_count": null, "id": "45", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "# Compare convergence behavior\n", "results = {\n", " \"Standard BO\": result_standard,\n", " \"BO with SDR\": result_sdr\n", "}\n", "\n", "# SDR typically converges faster than standard BO\n", "fig = om.criterion_plot(results)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "46", "metadata": {}, "source": [ "## Gaussian Process Configuration" ] }, { "cell_type": "markdown", "id": "47", "metadata": {}, "source": [ "`\"bayesian-optimization\"` uses a Gaussian Process (GP) as the surrogate model. Its behavior can be tuned with these options via algo_options:\n", "\n", "\n", "* **alpha**: noise level in function evaluations\n", "\n", " * lower values (e.g.,`1e-6`): assumes nearly precise function evaluations\n", " * higher values (e.g., `1e-2`): assumes noisy evaluations\n", "\n", "* **n\\_restarts**: Number of times to restart the optimization.\n", "\n", "* **seed** → ensures reproducible results.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "48", "metadata": { "vscode": { "languageId": "plaintext" } }, "outputs": [], "source": [ "algo_options = {\n", " \"alpha\": 1e-3,\n", " \"n_restarts\": 5,\n", " \"seed\": 42,\n", "}\n", "\n", "result_configured = om.minimize(\n", " fun=sphere,\n", " params=np.array([3.0, 3.0]),\n", " algorithm=\"bayes_opt\",\n", " bounds=om.Bounds(lower=np.full(2, -5.0), upper=np.full(2, 5.0)),\n", " algo_options=algo_options\n", ")\n", "\n", "print(\"Configured GP results:\")\n", "print(f\" Best value: {result_configured.fun}\")\n", "print(f\" Function evaluations: {result_configured.n_fun_evals}\")" ] }, { "cell_type": "markdown", "id": "49", "metadata": {}, "source": [ "## Summary\n", "\n", "Bayesian optimization is a powerful tool for optimizing expensive black-box functions. Key takeaways:\n", "\n", "1. **Choose the right acquisition function** based on your exploration/exploitation needs\n", "2. **Tune acquisition parameters** like kappa (UCB) or xi (EI) to control the trade-off\n", "3. **Use SDR** for high-dimensional problems to focus the search\n", "4. **Configure the GP properly** with appropriate noise levels and restarts\n", "\n", "For more detailed information, check out the [bayesian-optimization documentation](https://bayesian-optimization.github.io/BayesianOptimization/3.1.0/index.html#)." ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 } (how-to)= # How-to Guides How-to Guides show how to achieve specific tasks. In many cases they show you how to use advanced options. For a more basic introduction, check out the [tutorials](tutorials). ```{toctree} --- maxdepth: 1 --- how_to_criterion_function how_to_start_parameters how_to_derivatives how_to_specify_algorithm_and_algo_options how_to_algorithm_selection how_to_bounds how_to_constraints how_to_globalization how_to_multistart how_to_visualize_histories how_to_change_plotting_backend how_to_scaling how_to_logging how_to_errors_during_optimization how_to_slice_plot how_to_benchmarking how_to_add_optimizers how_to_document_optimizers ``` { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(how-to-fun)=\n", "\n", "# How to write objective functions\n", "\n", "optimagic is very flexible when it comes to the objective function and its derivatives. \n", "In this how-to guide we start with simple examples, that would also work with \n", "scipy.optimize before we show advanced options and their advantages. \n", "\n", "## The simplest case\n", "\n", "In the simplest case, `fun` maps a numpy array into a scalar objective value. The name\n", "of first argument of `fun` is arbitrary. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "import optimagic as om\n", "\n", "\n", "def sphere(x):\n", " return x @ x\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(3),\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "res.params.round(6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## More flexible `params`\n", "\n", "In all but the most simple problems, a flat numpy array is not ideal to keep track of \n", "all the different parameters one wants to optimize over. Therefore, optimagic accepts \n", "objective functions that work with other parameter formats. Below we show a simple \n", "example. More examples can be found [here](how_to_start_parameters.md).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def dict_fun(x):\n", " return x[\"a\"] ** 2 + x[\"b\"] ** 4\n", "\n", "\n", "res = om.minimize(\n", " fun=dict_fun,\n", " params={\"a\": 1, \"b\": 2},\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "\n", "res.params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The important thing is that the `params` provided to `minimize` need to have the format \n", "that is expected by the objective function.\n", "\n", "## Functions with additional arguments\n", "\n", "In many applications, the objective function takes more than `params` as argument. \n", "This can be achieved via `fun_kwargs`. Take the following simplified example:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def shifted_sphere(x, offset):\n", " return (x - offset) @ (x - offset)\n", "\n", "\n", "res = om.minimize(\n", " fun=shifted_sphere,\n", " params=np.arange(3),\n", " algorithm=\"scipy_lbfgsb\",\n", " fun_kwargs={\"offset\": np.ones(3)},\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`fun_kwargs` is a dictionary with keyword arguments for `fun`. There is no constraint\n", "on the number or names of those arguments." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Least-Squares problems\n", "\n", "Many estimation problems have a least-squares structure. If so, specialized optimizers that exploit this structure can be much faster than standard optimizers. The `sphere` function from above is the simplest possible least-squarse problem you could imagine: the least-squares residuals are just the params. \n", "\n", "To use least-squares optimizers in optimagic, you need to mark your function with \n", "a decorator and return the least-squares residuals instead of the aggregated function value.\n", "\n", "For a short explanation of scalar, least-squares, and likelihood problem types\n", "(`AggregationLevel`) and what each decorator expects, see\n", "{ref}`aggregation_level`.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "@om.mark.least_squares\n", "def ls_sphere(params):\n", " return params\n", "\n", "\n", "res = om.minimize(\n", " fun=ls_sphere,\n", " params=np.arange(3),\n", " algorithm=\"pounders\",\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Any least-squares optimization problem is also a standard optimization problem. You \n", "can therefore optimize least-squares functions with scalar optimizers as well:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=ls_sphere,\n", " params=np.arange(3),\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Returning additional information\n", "\n", "You can return additional information such as intermediate results, debugging information, etc. in your objective function. This information will be stored in a database if you use [logging](how_to_logging.ipynb).\n", "\n", "To do so, you need to return a `FunctionValue` object." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere_with_info(x):\n", " return om.FunctionValue(value=x @ x, info={\"avg\": x.mean()})\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere_with_info,\n", " params=np.arange(3),\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "\n", "res.params.round(6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `info` can be an arbitrary dictionary. In the oversimplified example we returned the \n", "mean of the parameters, which could have been recovered from the params history that \n", "is collected anyways but in real applications this feature can be helpful. " ] } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 2 } { "cells": [ { "cell_type": "markdown", "metadata": { "vscode": { "languageId": "plaintext" } }, "source": [ "# How to add optimizers to optimagic\n", "\n", "This is a hands-on guide that shows you how to use custom optimizers with optimagic or\n", "how to contribute an optimizer to the optimagic library.\n", "\n", "We have many [examples of optimizers](https://github.com/optimagic-dev/optimagic/tree/main/src/optimagic/optimizers) that are already part of optimagic and you can learn a lot from looking at \n", "those. However, only looking at the final results might be a bit intimidating and does\n", "not show the process of exploring a new optimizer library and gradually developing a \n", "wrapper. \n", "\n", "This guide is there to fill the gap. It tells the story of how the `pygmo_gaco`\n", "optimizer was added to optimagic by someone who was unfamiliar with pygmo or the \n", "gaco algorithm. \n", "\n", "The steps of adding an algorithm are roughly as follows:\n", "\n", "1. **Understand how to use the algorithm**: Play around with the algorithm you want to \n", "add in a notebook and solve some simple problems with it. Only move on to the next step \n", "after you have a solid understanding. This is completely unrelated to optimagic and only\n", "about he algorithm implementation you want to wrap. \n", "2. **Understand how the algorithm works**: Read documentation,\n", "research papers and other resources to find out why this algorithm was created and what \n", "problems it is supposed to solve really well. \n", "3. **Implement the minimal wrapper**: Learn about the `om.mark.minimizer` decorator as \n", "well as the `om.InternalOptimizationProblem` and the `om.Algorithm` classes. Implement a \n", "minimal version of your wrapper and test it.\n", "4. **Complete and refactor the wrapper**: Make sure that all convergence criteria, \n", "stopping criteria, and tuning parameters the algorithm supports can be passed to your \n", "wrapper. Also check that the algorithm gets everything it needs to achieve maximum \n", "performance (e.g. closed form derivatives and batch function evaluators). Now is also \n", "the time to clean-up and refactor your code, especially if you wrap multiple optimizers \n", "from a library.\n", "5. **Align the wrapper with optimagic conventions**: Use harmonized names wherever \n", "a convention exists. Think about good names everywhere else. Set stopping criteria \n", "similar to other optimizers and try to adhere to our [design philosophy](style_guide) \n", "when it comes to tuning parameters. \n", "6. **Integrate your code into optimagic**: Learn how to add an optional dependency to \n", "optimagic, where you need to put your code and how to add tests and documentation. \n", "\n", "\n", "## Gen AI Policy \n", "\n", "It is ok to use GenAI and AI based coding assistants to speed up the process of adding \n", "an optimizer to optimagic. They can be very useful for step 1 and 2. However, AI models \n", "often fail completely when filling out the arguments of `om.mark.minimizer`, when you \n", "ask them to come up with good names for tuning parameters or when you auto-generate the \n", "documentation. \n", "\n", "Even for step 1 and 2 you should not use an AI Model naively, but upload a paper or \n", "documentation page to provide context to the AI.\n", "\n", "Our policy is therefore:\n", "1. Only use AI for drafts that you double-check; Never rely on AI producing correct results \n", "2. Be transparent about your use of AI \n", "\n", "We will reject all Pull Requests that violate this policy. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Understand how to use the algorithm\n", "\n", "Understanding how to use an algorithm means that you are at least able to solve a \n", "simple optimization problem (like a sphere function or a rosenbrock function). \n", "\n", "The best starting point for this are usually tutorials or example notebooks from the \n", "documentation. An AI model can also be a good idea. \n", "\n", "The things you need to find out for any new algorithm are:\n", "\n", "1. How to code up the objective function \n", "2. How to run an optimization at default values\n", "3. How to pass tuning parameters \n", "4. How to pass bounds, constraints, derivatives, batch evaluators, etc. \n", "5. How to get results back from the optimizer\n", "\n", "### Objective functions in pygmo\n", "\n", "To add pygmo_gaco, let's start by looking at the pygmo [tutorials](https://esa.github.io/pygmo2/tutorials/tutorials.html). Objective functions are coded up via the Problem class. We skip using [pre-defined problems](https://esa.github.io/pygmo2/tutorials/using_problem.html) because they will not help us and directly go to [user defined problems](https://esa.github.io/pygmo2/tutorials/coding_udp_simple.html).\n", "\n", "The following is copied from the documentation:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pygmo as pg\n", "\n", "\n", "class sphere_function:\n", " def fitness(self, x):\n", " return [sum(x * x)]\n", "\n", " def get_bounds(self):\n", " return ([-1, -1], [1, 1])\n", "\n", "\n", "prob = pg.problem(sphere_function())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This looks simple enough. No subclassing is required, `fitness` implements the objective\n", "function, which returns the objective value as a list of a scalar and `get_bounds` returns \n", "the bounds. We can immediately see how we would adjust this for any scalar objective \n", "function. \n", "\n", "### How to run an optimization at default values\n", "\n", "After copy pasting from a few tutorials we find the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The initial population\n", "pop = pg.population(prob, size=20)\n", "# The algorithm; ker needs to be at most the population size to avoid errors\n", "algo = pg.algorithm(pg.gaco(ker=20))\n", "# The actual optimization process\n", "pop = algo.evolve(pop)\n", "# Getting the best individual in the population\n", "best_fitness = pop.get_f()[pop.best_idx()]\n", "print(best_fitness)\n", "best_x = pop.get_x()[pop.best_idx()]\n", "print(np.round(best_x, 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It looks like the optimization worked, even though the precision is not great. The true optimal function value is 0 and the true optimal parameters are [0, 0]. But global algorithms like gaco are almost never precise, so this is good enough. \n", "\n", "We can also see that pygmo is really organized around concepts that are specific to genetic optimizers. Examples are `population` and `evolve`. The optimagic wrapper will hide the details (i.e. users don't have to create a population) but still allow full customization (the population size will be an algorithm specific option that can be set by the user).\n", "\n", "### How to pass tuning parameters\n", "\n", "We already saw in the previous step that tuning parameters like `ker` are passed when the \n", "algorithm is created. \n", "\n", "All supported tuning parameters of gaco are listed and described \n", "[here](https://esa.github.io/pygmo2/algorithms.html#pygmo.gaco). Unfortunately, the \n", "description is not great so we'll have to look into the [paper](https://digital.csic.es/bitstream/10261/54957/3/Extended_ant_colony_2009.pdf) for details. \n", "\n", "\n", "### How to pass bounds, constraints, derivatives, batch evaluators, etc. \n", "\n", "- We already saw how to pass bounds via the Problem class \n", "- gaco does not support any other constraints, so we don't need to pass them \n", "- gaco is derivative free, so we don't need to pass derivatives \n", "- gaco can parallelize, so we need to find out how to pass a batch version of the \n", "objective function\n", "\n", "After searching around in the pygmo documentation, we find out that our Problem needs to \n", "be extended with a [`batch_fitness`](https://esa.github.io/pygmo2/problem.html#pygmo.problem.batch_fitness)\n", "and our algorithm needs to know about [`pg.bfe()`](https://esa.github.io/pygmo2/bfe.html).\n", "In our previous example it will look like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pygmo as pg\n", "\n", "\n", "class sphere_function:\n", " def fitness(self, x):\n", " return [sum(x * x)]\n", "\n", " def get_bounds(self):\n", " return ([-1, -1], [1, 1])\n", "\n", " # dvs represents a batch of parameter vectors at which the objective function is\n", " # evaluated. However it is stored in an unintuitive format that needs to be reshaped\n", " # to get at the actual parameter vectors.\n", " def batch_fitness(self, dvs):\n", " dim = len(self.get_bounds()[0])\n", " x_list = list(dvs.reshape(-1, dim))\n", " # we don't actually need to parallelize to find out how batch evaluators work\n", " # and optimagic will make it really easy to parallelize this later on.\n", " eval_list = [self.fitness(x)[0] for x in x_list]\n", " evals = np.array(eval_list)\n", " return evals\n", "\n", "\n", "prob = pg.problem(sphere_function())\n", "\n", "pop = pg.population(prob, size=20)\n", "\n", "# creating the algorithm now requires 3 steps\n", "pygmo_uda = pg.gaco(ker=20)\n", "pygmo_uda.set_bfe(pg.bfe())\n", "algo = pg.algorithm(pygmo_uda)\n", "\n", "pop = algo.evolve(pop)\n", "best_fitness = pop.get_f()[pop.best_idx()]\n", "print(best_fitness)\n", "best_x = pop.get_x()[pop.best_idx()]\n", "print(np.round(best_x, 4))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For this how-to guide we leave it at this basic exploration of the pygmo library. If you actually contributed an optimizer to optimagic, you would have to explore much more and document your exploration to convince us that you understand the library you wrap in detail. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### How to get results back \n", "\n", "The results are stored as part of the evolved population" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(\"Best function value: \", pop.get_f()[pop.best_idx()][0])\n", "print(\"Best parameters: \", pop.get_x()[pop.best_idx()])\n", "print(\"Number of function evaluations: \", pop.problem.get_fevals())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Understand how the algorithm works\n", "\n", "Here we want to find out as much as possible about the algorithm. Common questions \n", "that should be answered are:\n", "- For which kind of problems and situations was it designed?\n", "- How does it work (intuitively)?\n", "- Are there any papers, blogposts or other sources of information on the algorithm? \n", "- Which tuning parameters does it have and what do they mean? \n", "- Are there known limitations? \n", "\n", "### For which kind of problems and situations was it desigend \n", "\n", "gaco is a global optimizer that does not use derivative information. It should not be\n", "used if you only need a local optimum or if you have derivatives. Other algorithms would \n", "be more efficient and more precise there. \n", "\n", "Since gaco can evaluate the objective function in parallel it is designed for problems \n", "with expensive objective functions. \n", "\n", "\n", "### How does it work (intuitively)\n", "\n", "Ant colony optimization is a class of optimization algorithms modeled on the\n", "actions of an ant colony. Artificial \"ants\" (e.g. simulation agents) locate\n", "optimal solutions by moving through a parameter space representing all\n", "possible solutions. Real ants lay down pheromones directing each other to\n", "resources while exploring their environment. The simulated \"ants\" similarly\n", "record their positions and the quality of their solutions, so that in later\n", "simulation iterations more ants locate better solutions.\n", "\n", "The generalized ant colony algorithm generates future generations of ants by\n", "using a multi-kernel gaussian distribution based on three parameters (i.e.,\n", "pheromone values) which are computed depending on the quality of each\n", "previous solution. The solutions are ranked through an oracle penalty\n", "method.\n", "\n", "\n", "### Are there any papers, blogposts or other sources of information on the algorithm? \n", "\n", "gaco was proposed in M. Schlueter, et al. (2009). Extended ant colony optimization for \n", "non-convex mixed integer non-linear programming. Computers & Operations Research.\n", "\n", "See [here](https://digital.csic.es/bitstream/10261/54957/3/Extended_ant_colony_2009.pdf) for a free pdf. \n", "\n", "### Which tuning parameters does it have and what do they mean? \n", "\n", "The following is not just copied from the documentation but extended by reading the\n", "paper. It is super important to provide as much information as possible for every \n", "tunig parameter: \n", "\n", "- gen (int): number of generations.\n", "- ker (int): number of solutions stored in the solution archive. Must be <= the population\n", " size. \n", "- q (float): convergence speed parameter. This parameter manages the convergence speed\n", " towards the found minima (the smaller the faster). It must be positive and can be\n", " larger than 1. The default is 1.0 until **threshold** is reached. Then it\n", " is set to 0.01.\n", "- oracle (float): oracle parameter used in the penalty method.\n", "- acc (float): accuracy parameter for maintaining a minimum penalty\n", " function's values distances.\n", "- threshold (int): when the iteration counter reaches the threshold the\n", " convergence speed is set to 0.01 automatically. To deactivate this effect\n", " set the threshold to stopping.maxiter which is the largest allowed\n", " value.\n", "- n_gen_mark (int): parameter that determines the convergence speed of the standard \n", " deviations. This must be an integer.\n", "- impstop (int): if a positive integer is assigned here, the algorithm will count the \n", " runs without improvements, if this number exceeds the given value, the algorithm \n", " will be stopped.\n", "- evalstop (int): maximum number of function evaluations.\n", "- focus (float): this parameter makes the search for the optimum greedier\n", " and more focused on local improvements (the higher the greedier). If the\n", " value is very high, the search is more focused around the current best\n", " solutions. Values larger than 1 are allowed.\n", "- memory (bool): if True, memory is activated in the algorithm for multiple calls.\n", "- seed (int): seed used by the internal random number generator (default is random).\n", "\n", "\n", "### Are there known limitations \n", "\n", "No. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Implement the minimal wrapper\n", "\n", "\n", "### Learn the relevant functions and classes\n", "\n", "Before you implement a minimal wrapper, you need to familiarize yourself with a few\n", "important [classes and functions](functions_and_classes_for_internal_optimizers) \n", "you will need. \n", "\n", "- The `mark.miminizer` decorator \n", "- The `Algorithm` class \n", "- The `InternalOptimizationProblem` class \n", "- The `InternalOptimizeResult` class \n", "\n", "**Your task will be to subclass `Algorithm`. Your subclass must be decorated with\n", "`mark.minizer` and override `Algorithm._solve_internal_problem`. `_solve_internal_problem`\n", "takes an `InternalOptimizationProblem` and returns an `InternalOptimizeResult`**\n", "\n", "```{note}\n", "Users of optimagic never create instances of `InternalOptimizationProblem` nor \n", "do they call the `_solve_internal_problem` methods of algorithms. Instead they call \n", "`minimize` or `maximize` which are much more convenient and flexible. \n", "\n", "`minimize` and `maximize` will then create an `InternalOptimizationProblem` from the \n", "user's inputs, call the `_solve_internal_problem` method and postprocess it to create an \n", "OptimizeResult. \n", "\n", "To summarize: The public `minimize` interface is optimized for user-friendliness. The \n", "`InternalOptimizeProblem` is optimized for easy wrapping of external libraries. \n", "```\n", "\n", "Below we define a heavily commented minimal version of a wrapper for pygmo's gaco \n", "algorithm. We stay as close as possible to the pygmo examples we have worked with \n", "before and ignore most tuning parameters for now. \n", "\n", "\n", "### Write the minimal implementation" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from dataclasses import dataclass\n", "\n", "from numpy.typing import NDArray\n", "\n", "import optimagic as om\n", "from optimagic.optimization.algorithm import Algorithm, InternalOptimizeResult\n", "from optimagic.optimization.internal_optimization_problem import (\n", " InternalOptimizationProblem,\n", ")\n", "from optimagic.typing import AggregationLevel, PositiveInt\n", "\n", "try:\n", " import pygmo as pg\n", "\n", " IS_PYGMO_INSTALLED = True\n", "except ImportError:\n", " IS_PYGMO_INSTALLED = False\n", "\n", "\n", "@om.mark.minimizer(\n", " # you can pick the name; convention is lowercase with underscores\n", " name=\"pygmo_gaco\",\n", " # the type of problem this optimizer can solve -> scalar problems; Other optimizers\n", " # solve likelihood or least_squares problems.\n", " solver_type=AggregationLevel.SCALAR,\n", " # is the optimizer available? -> only if pygmo is installed\n", " is_available=IS_PYGMO_INSTALLED,\n", " # is the optimizer a global optimizer? -> yes\n", " is_global=True,\n", " # does the optimizer need the jacobian? -> no, gaco is derivative free\n", " needs_jac=False,\n", " # does the optimizer need the hessian? -> no, gaco is derivative free\n", " needs_hess=False,\n", " # does the optimizer support parallelism? -> yes\n", " supports_parallelism=True,\n", " # does the optimizer support bounds? -> yes\n", " supports_bounds=True,\n", " # does the optimizer support linear constraints? -> no\n", " supports_linear_constraints=False,\n", " # does the optimizer support nonlinear constraints? -> no\n", " supports_nonlinear_constraints=False,\n", " # should the history be disabled? -> no\n", " disable_history=False,\n", ")\n", "# All algortihms need to be frozen dataclasses.\n", "@dataclass(frozen=True)\n", "class PygmoGaco(Algorithm):\n", " # for now only set one parameter to get things running. The rest will come later.\n", " stopping_maxiter: PositiveInt = 1000\n", " n_cores: int = 1\n", "\n", " def _solve_internal_problem(\n", " self, problem: InternalOptimizationProblem, x0: NDArray[np.float64]\n", " ) -> InternalOptimizeResult:\n", " # create a pygmo problem from the internal optimization problem\n", " # This is just slightly more abstract than before and actually simpler because\n", " # we have problem.batch_fun.\n", "\n", " n_cores = self.n_cores\n", "\n", " class PygmoProblem:\n", " def fitness(self, x):\n", " # problem.fun is not just the `fun` that was passed to `minimize` by\n", " # the user. It is a wrapper around fun with added error handling,\n", " # history collection, and reparametrization to enforce constraints.\n", " # Moreover, it always works on flat numpy arrays as parameters and\n", " # does not have additional arguments. The magic of optimagic is to\n", " # create this internal `fun` from the user's `fun`, so you don't have\n", " # to deal with constraints, weird parameter formats and similar when\n", " # implementing the wrapper.\n", " return [problem.fun(x)]\n", "\n", " def get_bounds(self):\n", " # problem.bounds is not just the `bounds` that was passed to `minimize`\n", " # by the user, which could have been a dictionary or some other non-flat\n", " # format. `problem.bounds` always contains flat arrays with lower and\n", " # upper bounds because this makes it easy to write wrappers.\n", " return (problem.bounds.lower, problem.bounds.upper)\n", "\n", " def batch_fitness(self, dvs):\n", " # The processing of dvs is pygmo specific.\n", " dim = len(self.get_bounds()[0])\n", " x_list = list(dvs.reshape(-1, dim))\n", " # problem.batch_fun is a parallelized version of problem.fun.\n", " eval_list = problem.batch_fun(x_list, n_cores)\n", " evals = np.array(eval_list)\n", " return evals\n", "\n", " prob = pg.problem(PygmoProblem())\n", " pop = pg.population(prob, size=20)\n", " pygmo_uda = pg.gaco(ker=20)\n", " pygmo_uda.set_bfe(pg.bfe())\n", " algo = pg.algorithm(pygmo_uda)\n", " pop = algo.evolve(pop)\n", " best_fun = pop.get_f()[pop.best_idx()][0]\n", " best_x = pop.get_x()[pop.best_idx()]\n", " n_fun_evals = pop.problem.get_fevals()\n", " # For now we only use a few fields of the InternalOptimizeResult.\n", " out = InternalOptimizeResult(\n", " x=best_x,\n", " fun=best_fun,\n", " n_fun_evals=n_fun_evals,\n", " )\n", " return out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Test the minimal wrapper directly\n", "\n", "So now that we have a wrapper, what do we do with it? And how can we be sure it works?\n", "\n", "We'll first try it out directly with the `SphereExampleInternalOptimizationProblem`. \n", "This is only for debugging and testing purposes. A user would never create an \n", "InternalOptimizationProblem and call an algorithm with it. It's called \"Internal\" for \n", "a reason!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from optimagic.optimization.internal_optimization_problem import (\n", " SphereExampleInternalOptimizationProblem,\n", ")\n", "\n", "problem = SphereExampleInternalOptimizationProblem()\n", "\n", "gaco = PygmoGaco()\n", "\n", "result = gaco._solve_internal_problem(problem, x0=np.array([1.0, 1.0]))\n", "\n", "print(result.fun)\n", "print(result.x)\n", "print(result.n_fun_evals)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Use the minimal wrapper in minimize\n", "\n", "The internal testing gives us some confidence that the wrapper works correctly and would \n", "have been good for debugging if it didn't. But now we want to test the wrapper in the\n", "way it would be used later: via `minimize`\n", "\n", "With this we also get all the benefits of optimagic, from history collection and \n", "criterion plots to flexible parameter formats. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=lambda x: x @ x,\n", " params=np.arange(5),\n", " algorithm=PygmoGaco,\n", " bounds=om.Bounds(lower=-np.ones(5), upper=np.ones(5)),\n", ")\n", "\n", "om.criterion_plot(res, monotone=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4 Complete and refactor the wrapper\n", "\n", "To keep things simple, we left out almost all tuning parameters of the gaco algorithm \n", "when we wrote the minimal wrapper. \n", "\n", "Now it's time to add them. You can add them one by one and make sure nothing breaks by \n", "testing your wrapper after each change - both with the internal problem and via \n", "minimize. \n", "\n", "Moreover, our code looks quite messy currently. Despite being a minimal wrapper, the \n", "`_solve_internal_problem` method is quite long, unstructured and hard to read. \n", "\n", "The result of completing and refactoring the wrapper is too long to be repeated in the \n", "notebook. Instead you can look at the actual [implementation in optimagic](\n", "https://github.com/optimagic-dev/optimagic/blob/ba2678753587f91cea54de69ff76cb3dcb4257d4/src/optimagic/optimizers/pygmo_optimizers.py#L70)\n", "\n", "\n", "The PygmoGaco class now contains all tuning parameters we identified in step 2 as\n", "dataclass fields. They all have very useful type-hints that don't just show whether\n", "a parameter is an int, str or float but also which values it can take (e.g. PositiveInt).\n", "\n", "`_solve_internal_problem` is now also much cleaner. It mainly maps our mor descriptive \n", "names of tuning parameters to the old pygmo names and then calls a function called \n", "`_minimize_pygmo` that does all the heavy lifting and can be re-used for other pygmo \n", "optimizers. \n", "\n", "The arguments to `mark.minimizer` have not changed. They always need te be set correctly,\n", "even for minimal working examples. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 5. Align the wrapper with optimagic conventions\n", "\n", "To make switching between different algorithm as simple as possible, we align the names \n", "of commonly used convergence and stopping criteria. We also align the default values for \n", "stopping and convergence criteria as much as possible. \n", "\n", "You can find the harmonized names and value [here](algo_options_docs). \n", "\n", "To align the names of other tuning parameters as much as possible with what is already \n", "there, simple have a look at the optimizers we already wrapped. For example, if you are \n", "wrapping a bfgs or lbfgs algorithm from some libray, try to look at all existing wrappers \n", "of bfgs algorithms and use the same names for the same options. \n", "\n", "You can see what this means for the gaco algorithm [here](\n", "https://github.com/optimagic-dev/optimagic/blob/ba2678753587f91cea54de69ff76cb3dcb4257d4/src/optimagic/optimizers/pygmo_optimizers.py#L70)\n", "\n", "In the future we will provide much more extensive guidelines for harmonization. \n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": "## 6. Integrate your code into optimagic\n\nSo far you could have worked in a Jupyter Notebook. Integrating your code into\noptimagic only requires a few small changes:\n\n1. Add new dependencies to the `[tool.pixi.feature.test.dependencies]` section of\n`pyproject.toml` and run `pixi install` to update the lock file. Then re-create the\nenvironment to make sure that the environment is the same as we will use for continuous\nintegration. If your dependencies don't work on all platforms (e.g. linux only packages),\nskip this entire step and reach out to a core contributor for help.\n2. Save the code for your algorithm wrapper in a .py file in `optimagic.algorithms`.\nUse an existing file if you wrap another algorithm from a library we already had.\nOtherwise, create a new file.\n3. Run `pre-commit run --all-files`. This will trigger an automatic code generation\nthat fully integrates your wrapper into our algorithm selection tool.\n4. Run `pytest`. This will run at least a few tests for your new algorithm. Add more\ntests for algorithm specific things (e.g. tests that make sure tuning parameters have\nthe intended effects).\n5. Write documentation. The documentation should contain everything you figured out in\nstep 2. You can either write it into the docstring of your algorithm class (preferred,\nas this is what we will do for all algorithms in the long run) or in `algorithms.md`\nin the documentation.\n6. Create a pull request [in the optimagic repository](https://github.com/optimagic-dev/optimagic)\nand ask for a review." } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.15" } }, "nbformat": 4, "nbformat_minor": 2 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(how-to-select-algorithms)=\n", "# How to select a local optimizer\n", "\n", "This guide explains how to choose a local optimizer that works well for your problem. \n", "Depending on your [strategy for global optimization](how_to_globalization.ipynb) it \n", "is also relevant for global optimization problems. \n", "\n", "## Important facts \n", "\n", "- There is no optimizer that works well for all problems \n", "- Making the right choice can lead to enormous speedups\n", "- Making the wrong choice can mean that you [don't solve your problem at all](algo-selection-how-important). Sometimes,\n", "optimizers fail silently!\n", "\n", "\n", "## The three steps for selecting algorithms\n", "\n", "Algorithm selection is a mix of theory and experimentation. We recommend the following \n", "steps:\n", "\n", "1. **Theory**: Based on the properties of your problem, start with 3 to 5 candidate algorithms. \n", "You may use the decision tree below.\n", "2. **Experiments**: Run the candidate algorithms for a small number of function \n", "evaluations and compare the results in a *criterion plot*. As a rule of thumb, use \n", "between `n_params` and `10 * n_params` evaluations. \n", "3. **Optimization**: Re-run the algorithm with the best results until \n", "convergence. Use the best parameter vector from the experiments as start parameters.\n", "\n", "We will walk you through the steps in an [example](algo-selection-example-problem)\n", "below. These steps work well for most problems but sometimes you need \n", "[variations](algo-selection-steps-variations).\n", "\n", "\n", "## A decision tree \n", "\n", "This is a practical guide for narrowing down the set of algorithms to experiment with:\n", "\n", "```{mermaid}\n", "graph LR\n", " classDef highlight fill:#FF4500;\n", " A[\"Do you have
nonlinear
constraints?\"] -- yes --> B[\"differentiable?\"]\n", " B[\"Is your objective function differentiable?\"] -- yes --> C[\"ipopt
nlopt_slsqp
scipy_trust_constr\"]\n", " B[\"differentiable?\"] -- no --> D[\"scipy_cobyla
nlopt_cobyla\"]\n", "\n", " A[\"Do you have
nonlinear constraints?\"] -- no --> E[\"Can you exploit
a least-squares
structure?\"]\n", " E[\"Can you exploit
a least-squares
structure?\"] -- yes --> F[\"differentiable?\"]\n", " E[\"Can you exploit
a least-squares
structure?\"] -- no --> G[\"differentiable?\"]\n", "\n", " F[\"differentiable?\"] -- yes --> H[\"scipy_ls_lm
scipy_ls_trf
scipy_ls_dogbox\"]\n", " F[\"differentiable?\"] -- no --> I[\"nag_dflos
pounders
tao_pounders\"]\n", "\n", " G[\"differentiable?\"] -- yes --> J[\"scipy_lbfgsb
nlopt_lbfgsb
fides\"]\n", " G[\"differentiable?\"] -- no --> K[\"nlopt_bobyqa
nlopt_neldermead
neldermead_parallel\"]\n", "\n", "```\n", "\n", "Going through the different questions will give you a list of candidate algorithms. \n", "All algorithms in that list are designed for the same problem class but use different \n", "approaches to solve the problem. Which of them works best for your problem can only be \n", "found out through experimentation.\n", "\n", "```{note}\n", "Many books on numerical optimization focus strongly on the inner workings of algorithms.\n", "They will, for example, describe the difference between a trust-region algorithm and a \n", "line-search algorithm in a lot of detail. We have an [intuitive explanation](../explanation/explanation_of_numerical_optimizers.md) of this too. Understanding these details is important for configuring and\n", "troubleshooting optimizations, but not for algorithm selection. For example, If you have\n", "a scalar, differentiable problem without nonlinear constraints, the decision tree \n", "suggests `fides` and two variants of `lbfgsb`. `fides` is a trust-region algorithm, \n", "`lbfgsb` is a line-search algorithm. Both are designed to solve the same kinds of \n", "problems and which one works best needs to be found out through experimentation.\n", "```\n", "\n", "## Filtering algorithms \n", "\n", "An even more fine-grained version of the decision tree is built into optimagic's \n", "algorithm selection tool, which can filter algorithms based on the properties of \n", "your problem. To make this concrete, assume we are looking for a **local** optimizer for \n", "a **differentiable** problem with a **scalar** objective function and \n", "**bound constraints**. \n", "\n", "To find all algorithms that match our criteria, we can simply type:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import optimagic as om\n", "\n", "om.algos.Local.GradientBased.Scalar.Bounded.All" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The available filters are: GradientBased, GradientFree, Global, Local, Bounded, \n", "LinearConstrained, NonlinearConstrained, Scalar, LeastSquares, Likelihood, and Parallel.\n", "You can apply them in any order your want. They are also discoverable, i.e. the \n", "autocomplete feature of your editor will show you all filters you can apply on top of \n", "your current selection.\n", "\n", "Using `.All` after applying filters shows you all algorithms optimagic knows of that \n", "satisfy your criteria. Some of them require optional dependencies. To show only the \n", "algorithms that are available with the packages you have currently installed, use \n", "`.Available` instead of `.All`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An even more fine-grained way of filtering is described in [Filtering Algorithms Using Bounds](filtering_algorithms_using_bounds)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(algo-selection-example-problem)=\n", "\n", "## An example problem\n", "\n", "As an example we use the [Trid function](https://www.sfu.ca/~ssurjano/trid.html). The Trid function has no local minimum except \n", "the global one. It is defined for any number of dimensions, we will pick 20. As starting \n", "values we will pick the vector [0, 1, ..., 19]. \n", "\n", "A Python implementation of the function and its gradient looks like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "\n", "def trid_scalar(x):\n", " \"\"\"Implement Trid function: https://www.sfu.ca/~ssurjano/trid.html.\"\"\"\n", " return ((x - 1) ** 2).sum() - (x[1:] * x[:-1]).sum()\n", "\n", "\n", "def trid_gradient(x):\n", " \"\"\"Calculate gradient of trid function.\"\"\"\n", " l1 = np.insert(x, 0, 0)\n", " l1 = np.delete(l1, [-1])\n", " l2 = np.append(x, 0)\n", " l2 = np.delete(l2, [0])\n", " return 2 * (x - 1) - l1 - l2" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Theory\n", "\n", "\n", "\n", "Let's go through the decision tree for the Trid function:\n", "\n", "1. **No** nonlinear constraints our solution needs to satisfy\n", "2. **No** least-squares structure we can exploit \n", "3. **Yes**, the function is differentiable. We even have a closed form gradient that \n", "we would like to use. \n", "\n", "We therefore end up with the candidate algorithms `scipy_lbfgsb`, `nlopt_lbfgsb`, and \n", "`fides`.\n", "\n", "```{note}\n", "If your function is differentiable but you do not have a closed form gradient (yet), \n", "we suggest to use at least one gradient based optimizer and one gradient free optimizer.\n", "in your experiments. Optimagic will use numerical gradients in that case. For details, \n", "see [here](how_to_derivatives.ipynb).\n", "```\n", "\n", "\n", "### Step 2: Experiments\n", "\n", "To find out which algorithms work well for our problem, we simply run optimizations with\n", "all candidate algorithms in a loop and store the result in a dictionary. We limit the \n", "number of function evaluations to 8. Since some algorithms only support a maximum number\n", "of iterations as stopping criterion we also limit the number of iterations to 8.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results = {}\n", "for algo in [\"scipy_lbfgsb\", \"nlopt_lbfgsb\", \"fides\"]:\n", " results[algo] = om.minimize(\n", " fun=trid_scalar,\n", " jac=trid_gradient,\n", " params=np.arange(20),\n", " algorithm=algo,\n", " algo_options={\"stopping_maxfun\": 8, \"stopping_maxiter\": 8},\n", " )\n", "\n", "fig = om.criterion_plot(results, max_evaluations=8)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All optimizers work pretty well here and since this is a very simple problem, any of them \n", "would probably find the optimum in a reasonable time. However, `nlopt_lbfgsb` is a bit \n", "better than the others, so we will select it for the next step. In more difficult\n", "examples, the difference between optimizers can be much more pronounced.\n", "\n", "### Step 3: Optimization \n", "\n", "All that is left to do is to run the optimization until convergence with the best \n", "optimizer. To avoid duplicated calculations, we can already start from the previously \n", "best parameter vector:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "best_x = results[\"nlopt_lbfgsb\"].params\n", "results[\"nlopt_lbfgsb_complete\"] = om.minimize(\n", " fun=trid_scalar,\n", " jac=trid_gradient,\n", " params=best_x,\n", " algorithm=\"nlopt_lbfgsb\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Looking at the result in a criterion plot we can see that the optimizer converges after \n", "a bit more than 30 function evaluations. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(results)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(algo-selection-steps-variations)=\n", "\n", "## Variations of the four steps\n", "\n", "The four steps described above work very well in most situations. However, sometimes \n", "it makes sense to deviate: \n", "\n", "- If you are unsure about some of the questions in step 1, select more algorithms for \n", "the experimentation phase and run more than 1 algorithm until convergence. \n", "- If it is very important to find a precise optimum, run more than 1 algorithm until \n", "convergence. \n", "- If you have a very fast objective function, simply run all candidate algorithms until \n", "convergence. \n", "- If you have a differentiable objective function but no closed form derivative, use \n", "at least one gradient based optimizer and one gradient free optimizer in the \n", "experiments. See [here](how_to_derivatives.ipynb) to learn more about derivatives.\n", "\n", "\n", "(algo-selection-how-important)=\n", "\n", "## How important was it?\n", "\n", "The Trid function is differentiable and very well behaved in almost every aspect. \n", "Moreover, it has a very short runtime. One would think that any optimizer can find its \n", "optimum. So let's compare the selected optimizer with a few others:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results = {}\n", "for algo in [\"nlopt_lbfgsb\", \"scipy_neldermead\", \"scipy_cobyla\"]:\n", " results[algo] = om.minimize(\n", " fun=trid_scalar,\n", " jac=trid_gradient,\n", " params=np.arange(20),\n", " algorithm=algo,\n", " )\n", "\n", "fig = om.criterion_plot(results)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that our chosen optimizer solves the problem with less than 35 function \n", "evaluations. At this point, the two gradient-free optimizers have not yet made \n", "significant progress. CoByLA gets reasonably close to an optimum after about 4k \n", "evaluations. Nelder-Mead gets stuck after 8k evaluations and fails to solve the problem. \n", "\n", "This example shows not only that the choice of optimizer is important but that the commonly \n", "held belief that gradient free optimizers are generally more robust than gradient based \n", "ones is dangerous! The Nelder-Mead algorithm did \"converge\" and reports success, but\n", "did not find the optimum. It did not even get stuck in a local optimum because we know \n", "that the Trid function does not have local optima except the global one. It just got \n", "stuck somewhere. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results[\"scipy_neldermead\"].success" ] } ], "metadata": { "kernelspec": { "display_name": "optimagic-docs", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.16" } }, "nbformat": 4, "nbformat_minor": 4 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# How to Benchmark Optimization Algorithms\n", "\n", "Benchmarking optimization algorithms is an important step when developing a new algorithm or when searching for an algorithm that is good at solving a particular problem. \n", "\n", "In general, benchmarking constists of the following steps:\n", "\n", "1. Define the test problems (or get pre-implemented ones)\n", "2. Define the optimization algorithms and the tuning parameters you want to try\n", "3. Run the benchmark\n", "4. Plot the results\n", "\n", "optimagic helps you with all of these steps!" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## 1. Get Test Problems\n", "\n", "optimagic includes the problems of [Moré and Wild (2009)](https://doi.org/10.1137/080724083) as well as [Cartis and Roberts](https://arxiv.org/abs/1710.11005).\n", "\n", "Each problem consist of the `inputs` (the criterion function and the start parameters) and the `solution` (the optimal parameters and criterion value) and optionally provides more information.\n", "\n", "Below we load a subset of the Moré and Wild problems and look at one particular Rosenbrock problem that has difficult start parameters." ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [], "source": [ "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "problems = om.get_benchmark_problems(\"example\")" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## 2. Specify the Optimizers\n", "\n", "To select optimizers you want to benchmark on the set of problems, you can simply specify them as a list. Advanced examples - that do not only compare algorithms but also vary the `algo_options` - can be found below. " ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "optimizers = [\n", " \"nag_dfols\",\n", " \"scipy_neldermead\",\n", " \"scipy_truncated_newton\",\n", "]" ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ "## 3. Run the Benchmark\n", "\n", "Once you have your problems and your optimizers set up, you can simply use `run_benchmark`. The results are a dictionary with one entry for each (problem, algorithm) combination. Each entry not only saves the solution but also the history of the algorithm's criterion and parameter history. " ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [], "source": [ "results = om.run_benchmark(\n", " problems,\n", " optimizers,\n", ")" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "## 4a. Profile plots\n", "\n", "**Profile Plots** compare optimizers over a whole problem set. \n", "\n", "The literature distinguishes **data profiles** and **performance profiles**. Data profiles use a normalized runtime measure whereas performance profiles use an absolute one. The profile plot does not normalize runtime by default. To do this, simply set `normalize_runtime` to True. For background information, check [Moré and Wild (2009)](https://doi.org/10.1137/080724083). " ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "fig = om.profile_plot(\n", " problems=problems,\n", " results=results,\n", ")\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ ":::{note}\n", "\n", "For details on using other plotting backends, see [How to change the plotting backend](how_to_change_plotting_backend.ipynb).\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "The x axis shows runtime per problem. The y axis shows the share of problems each algorithm solved within that runtime. Thus, higher and further to the left values are desirable. Higher means more problems were solved and further to the left means, the algorithm found the solutions earlier. \n", "\n", "You can choose:\n", "\n", "- whether to use `n_evaluations` or `walltime` as **`runtime_measure`**\n", "- whether to normalize runtime such that the runtime of each problem is shown as a multiple of the fastest algorithm on that problem\n", "- how to determine when an evaluation is close enough to the optimum to be counted as converged. Convergence is always based on some measure of distance between the true solution and the solution found by an optimizer. Whether distiance is measured in parameter space, function space, or a combination of both can be specified. \n", "\n", "Below, we consider a problem to be solved if the distance between the parameters found by the optimizer and the true solution parameters are at most 0.1% of the distance between the start parameters and true solution parameters. " ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "fig = om.profile_plot(\n", " problems=problems,\n", " results=results,\n", " runtime_measure=\"n_evaluations\",\n", " stopping_criterion=\"x\",\n", " x_precision=0.001,\n", ")\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "## 4b. Convergence plots\n", "\n", "**Convergence Plots** look at particular problems and show the convergence of each optimizer on each problem. " ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "fig = om.convergence_plot(\n", " problems=problems,\n", " results=results,\n", " n_cols=2,\n", " problem_subset=[\"rosenbrock_good_start\", \"box_3d\"],\n", ")\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "The further to the left and the lower the curve of an algorithm, the better that algorithm performed.\n", "\n", "Often we are more interested in how close each algorithm got to the true solution in parameter space, not in criterion space as above. For this. we simply set the **`distance_measure`** to `parameter_space`. " ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "fig = om.convergence_plot(\n", " problems=problems,\n", " results=results,\n", " n_cols=2,\n", " problem_subset=[\"rosenbrock_good_start\", \"box_3d\"],\n", " distance_measure=\"parameter_distance\",\n", " stopping_criterion=\"x\",\n", ")\n", "\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## 5a. Convergence report\n", "\n", "The **Convergence Report** shows for each problem and optimizer which problems the optimizer solved successfully, failed to do so, or where it stopped with an error. The respective strings are \"success\", \"failed\", or \"error\".\n", "Moreover, the last column of the ```pd.DataFrame``` displays the number of dimensions of the benchmark problem." ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "df = om.convergence_report(\n", " problems=problems,\n", " results=results,\n", " stopping_criterion=\"y\",\n", " x_precision=1e-4,\n", " y_precision=1e-4,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "## 5b. Rank report\n", "\n", "The **Rank Report** shows the ranks of the algorithms for each problem; where 0 means the algorithm was the fastest on a given benchmark problem, 1 means it was the second fastest and so on. If an algorithm did not converge on a problem, the value is \"failed\". If an algorithm did encounter an error during optimization, the value is \"error\"." ] }, { "cell_type": "code", "execution_count": null, "id": "21", "metadata": {}, "outputs": [], "source": [ "df = om.rank_report(\n", " problems=problems,\n", " results=results,\n", " runtime_measure=\"n_evaluations\",\n", " stopping_criterion=\"y\",\n", " x_precision=1e-4,\n", " y_precision=1e-4,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "22", "metadata": {}, "outputs": [], "source": [ "df" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "## 5b. Traceback report\n", "\n", "The **Traceback Report** shows the tracebacks returned by the optimizers if they encountered an error during optimization. The resulting ```pd.DataFrame``` is empty if none of the optimizers terminated with an error, as in the example below." ] }, { "cell_type": "code", "execution_count": null, "id": "24", "metadata": {}, "outputs": [], "source": [ "df = om.traceback_report(problems=problems, results=results)" ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": {}, "outputs": [], "source": [ "df" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 5 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "(how-to-bounds)=\n", "\n", "# How to specify bounds\n", "\n", "## Constraints vs bounds \n", "\n", "optimagic distinguishes between bounds and constraints. Bounds are lower and upper bounds for parameters. In the literature, they are sometimes called box constraints. Examples for general constraints are linear constraints, probability constraints, or nonlinear constraints. You can find out more about general constraints in the next section on [How to specify constraints](how_to_constraints.md)." ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## Example objective function\n", "\n", "Let’s again look at the sphere function:" ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "import optimagic as om" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "def fun(x):\n", " return x @ x" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(fun=fun, params=np.arange(3), algorithm=\"scipy_lbfgsb\")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## Array params\n", "\n", "For params that are a `numpy.ndarray`, one can specify the lower and/or upper-bounds as an array of the same length.\n", "\n", "**Lower bounds**" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=fun,\n", " params=np.arange(3),\n", " bounds=om.Bounds(lower=np.ones(3)),\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "**Lower & upper-bounds**" ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=fun,\n", " params=np.arange(3),\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=om.Bounds(\n", " lower=np.array([-2, -np.inf, 1]),\n", " upper=np.array([-1, np.inf, np.inf]),\n", " ),\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## Pytree params\n", "\n", "Now let's look at a case where params is a more general pytree. We also update the sphere function by adding an intercept. Since the criterion always decreases when decreasing the intercept, there is no unrestricted solution. Lets fix a lower bound only for the intercept." ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [], "source": [ "params = {\"x\": np.arange(3), \"intercept\": 3}\n", "\n", "\n", "def fun(params):\n", " return params[\"x\"] @ params[\"x\"] + params[\"intercept\"]" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=fun,\n", " params=params,\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=om.Bounds(lower={\"intercept\": -2}),\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "optimagic tries to match the user provided bounds with the structure of params. This allows you to specify bounds for subtrees of params. In case your subtree specification results in an unidentified matching, optimagic will tell you so with a `InvalidBoundsError`. " ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "## params data frame\n", "\n", "It often makes sense to specify your parameters in a `pandas.DataFrame`, where you can utilize the multiindex for parameter naming. In this case, you can specify bounds as extra columns `lower_bound` and `upper_bound`.\n", "\n", "> **Note**\n", "> The columns are called `*_bound` instead of `*_bounds` like the argument passed to `minimize` or `maximize`. " ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "params = pd.DataFrame(\n", " {\"value\": [0, 1, 2, 3], \"lower_bound\": [0, 1, 1, -2]},\n", " index=pd.MultiIndex.from_tuples([(\"x\", k) for k in range(3)] + [(\"intercept\", 0)]),\n", ")\n", "params" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "def fun(params):\n", " x = params.loc[\"x\"][\"value\"].to_numpy()\n", " intercept = params.loc[\"intercept\"][\"value\"].iloc[0]\n", " value = x @ x + intercept\n", " return float(value)" ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun,\n", " params=params,\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "(filtering_algorithms_using_bounds)=\n", "\n", "## Filtering algorithms\n", "\n", "It is further possible to filter algorithms based on whether they support bounds, if bounds are required to run, and if infinite bounds are supported. The AlgoInfo class provides all information about the chosen algorithm, which can be accessed with algo.algo_info... . Suppose we are looking for a optimizer that supports bounds and strictly require them for the algorithm to run properly.\n", "\n", "To find all algorithms that support bounds and cannot run without bounds, we can simply do:\n" ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "from optimagic.algorithms import AVAILABLE_ALGORITHMS\n", "\n", "algos_with_bounds_support = [\n", " algo\n", " for name, algo in AVAILABLE_ALGORITHMS.items()\n", " if algo.algo_info.supports_bounds\n", "]\n", "my_selection = [\n", " algo for algo in algos_with_bounds_support if algo.algo_info.needs_bounds\n", "]\n", "my_selection[0:3]" ] }, { "cell_type": "markdown", "id": "19", "metadata": {}, "source": [ "Similarly, to find all algorithms that support infinite values in bounds , we can do:" ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [], "source": [ "my_selection2 = [\n", " algo\n", " for algo in algos_with_bounds_support\n", " if algo.algo_info.supports_infinite_bounds\n", "]\n", "my_selection2[0:3]" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "In case you you forget to specify bounds for a optimizer that strictly requires them or pass infinite values in bounds to a optimizer which does not support them, optimagic will raise an `IncompleteBoundsError`. " ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "## Coming from scipy" ] }, { "cell_type": "markdown", "id": "23", "metadata": {}, "source": [ "If `params` is a flat numpy array, you can also provide bounds in any format that \n", "is supported by [`scipy.optimize.minimize`](\n", "https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html). " ] } ], "metadata": { "kernelspec": { "display_name": "optimagic-docs", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 5 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# How to change the plotting backend" ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "optimagic supports various visualization libraries as plotting backends, which can be\n", "selected using the `backend` argument. In the following guide, we showcase the \n", "`criterion_plot` visualized using different backends." ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "import optimagic as om\n", "\n", "\n", "def sphere(x):\n", " return x @ x\n", "\n", "\n", "results = {}\n", "for algo in [\"scipy_lbfgsb\", \"scipy_neldermead\"]:\n", " results[algo] = om.minimize(sphere, params=np.arange(5), algorithm=algo)" ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## Backends" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "### Plotly" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "The default plotting library. To select the Plotly backend explicitly, set `backend=\"plotly\"`.\n", "\n", "The returned figure object is a [`plotly.graph_objects.Figure`](https://plotly.com/python-api-reference/generated/plotly.graph_objects.Figure.html).\n", "\n", "```{note}\n", "**Choose the Plotly renderer according to your environment:**\n", "\n", "- Use `plotly.io.renderers.default = \"notebook_connected\"` in Jupyter notebooks for interactive plots.\n", "- Use `plotly.io.renderers.default = \"browser\"` to open plots in your default web browser when running as a script.\n", "\n", "Refer to the [Plotly documentation](https://plotly.com/python/renderers/) for more details.\n", "```" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "fig = om.criterion_plot(results, backend=\"plotly\") # Also the default\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "### Matplotlib" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "To select the Matplotlib backend, set `backend=\"matplotlib\"`.\n", "\n", "The returned figure object is a [`matplotlib.axes.Axes`](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html).\n", "\n", "In case of grid plots (such as `convergence_plot` or `slice_plot`), the returned object is a 2-dimensional numpy array of `Axes` objects: [`numpy.ndarray`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.html)[[`matplotlib.axes.Axes`]](https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.html) of shape `(n_rows, n_cols)`." ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "ax = om.criterion_plot(results, backend=\"matplotlib\")" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "### Bokeh" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "To select the Bokeh backend, set `backend=\"bokeh\"`.\n", "\n", "The returned figure object is a [`bokeh.plotting.figure`](https://docs.bokeh.org/en/latest/docs/reference/plotting/figure.html).\n", "\n", "In case of grid plots (such as `convergence_plot` or `slice_plot`), the returned object is a [`bokeh.models.GridPlot`](https://docs.bokeh.org/en/latest/docs/reference/models/plots.html#bokeh.models.GridPlot) object.\n", "\n", "```{warning}\n", "- Bokeh applies themes globally. Passing the `template` parameter to a plotting function updates the theme for all existing and future Bokeh plots. If you do not pass `template`, a default template is applied, which will also change the global theme.\n", "- Bokeh doesn't support titles for grid plots. So, the `title` parameter in `slice_plot` is ignored when using the Bokeh backend.\n", "```\n" ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "from bokeh.io import output_notebook, show\n", "\n", "output_notebook()\n", "\n", "p = om.criterion_plot(results, backend=\"bokeh\")\n", "show(p)" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "### Altair" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "To select the Altair backend, set `backend=\"altair\"`.\n", "\n", "The returned figure object is an [`altair.Chart`](https://altair-viz.github.io/user_guide/generated/toplevel/altair.Chart.html).\n", "\n", "In case of grid plots (such as `convergence_plot` or `slice_plot`), the returned object is either an [`altair.Chart`](https://altair-viz.github.io/user_guide/generated/toplevel/altair.Chart.html) if there is only one subplot, an [`altair.HConcatChart`](https://altair-viz.github.io/user_guide/generated/toplevel/altair.HConcatChart.html) if there is only one row, or an [`altair.VConcatChart`](https://altair-viz.github.io/user_guide/generated/toplevel/altair.VConcatChart.html) otherwise.\n", "\n", "```{warning}\n", "Altair applies themes globally. Passing the `template` parameter to a plotting function updates the theme for all existing and future Altair plots. If you do not pass `template`, a default template is applied, which will also change the global theme.\n", "```\n", "\n", "```{note}\n", "It is mostly not required to set the renderer manually, as Altair automatically\n", "selects the appropriate renderer based on your environment. In this example,\n", "we explicitly set the renderer to ensure correct display within the documentation.\n", "\n", "Refer to the [Altair documentation](https://altair-viz.github.io/user_guide/display_frontends.html) for more details.\n", "```\n" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "import altair as alt\n", "\n", "# Setting the renderer is mostly not required. See note above.\n", "alt.renderers.enable(\"jupyter\")\n", "\n", "chart = om.criterion_plot(results, backend=\"altair\")\n", "chart.show()" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "## Customizing plots" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "Here, we provide a simple example of how to customize plots created with different backends.\n", "\n", "::::{tab-set}\n", "\n", ":::{tab-item} Plotly\n", "\n", "```python\n", "fig = om.criterion_plot(results, backend=\"plotly\")\n", "\n", "# Configure Axes\n", "fig.update_yaxes(title_text=\"Custom Y Label\", title_font_size=20)\n", "fig.update_xaxes(range=[0, 100])\n", "\n", "# Change legend position\n", "fig.update_layout(legend=dict(xanchor=\"left\", yanchor=\"top\", x=1, y=0.6))\n", "\n", "# Configure line properties\n", "# The index corresponding to a line, can be inferred from the legend\n", "# In case of criterion_plot, it is the order of optimizers in `results`\n", "fig.data[0].update(line=dict(width=4))\n", "fig.data[1].update(line=dict(dash=\"dashdot\"))\n", "\n", "fig.show()\n", "```\n", ":::\n", "\n", ":::{tab-item} Matplotlib\n", "\n", "```python\n", "ax = om.criterion_plot(results, backend=\"matplotlib\")\n", "\n", "# Configure Axis\n", "ax.set_ylabel(ylabel=\"Custom Y Label\", fontsize=20)\n", "ax.set_xlim(0, 100)\n", "\n", "# Change legend position\n", "ax.figure.legends[0].set_loc(\"outside center right\")\n", "\n", "# Configure line properties\n", "# The index corresponding to a line, can be inferred from the legend\n", "# In case of criterion_plot, it is the order of optimizers in `results`\n", "ax.lines[0].set_linewidth(4)\n", "ax.lines[1].set_linestyle(\"dashdot\")\n", "```\n", "\n", ":::\n", "\n", ":::{tab-item} Bokeh\n", "\n", "```python\n", "from bokeh.models import Range1d\n", "\n", "p = om.criterion_plot(results, backend=\"bokeh\")\n", "\n", "# Configure Axes\n", "p.yaxis.axis_label = \"Custom Y Label\"\n", "p.yaxis.axis_label_text_font_size = \"20pt\"\n", "p.x_range = Range1d(0, 100)\n", "\n", "# Change legend position\n", "p.add_layout(p.legend[0], \"right\")\n", "p.legend[0].location = \"center\"\n", "\n", "# Configure line properties\n", "# The index corresponding to a line, can be inferred from the legend\n", "# In case of criterion_plot, it is the order of optimizers in `results`\n", "p.renderers[0].glyph.line_width = 4\n", "p.renderers[1].glyph.line_dash = \"dashdot\"\n", "\n", "show(p)\n", "```\n", "\n", ":::\n", "\n", ":::{tab-item} Altair\n", "\n", "```{note}\n", "Due to the nature of Altair charts, top-level configuration may not work as expected. In these cases, it might be necessary to override the encoding.\n", "```\n", "\n", "```python\n", "import altair as alt\n", "\n", "chart = om.criterion_plot(results, backend=\"altair\")\n", "\n", "# Configure Axes\n", "chart = chart.encode(\n", " y=alt.Y(\"y\", axis=alt.Axis(title=\"Custom Y Label\", titleFontSize=20)),\n", " x=alt.X(\"x\", scale=alt.Scale(domain=(0, 100))),\n", ")\n", "\n", "# Configure lines\n", "chart = chart.encode(\n", " strokeWidth=alt.condition(\n", " alt.datum.name == \"scipy_lbfgsb\", alt.value(4), alt.value(2)\n", " ),\n", " strokeDash=alt.condition(\n", " alt.datum.name == \"scipy_neldermead\", alt.value([8, 4, 2, 4]), alt.value([1, 0])\n", " ),\n", ")\n", "\n", "chart.show()\n", "```\n", "\n", ":::\n", "\n", "::::" ] } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.17" } }, "nbformat": 4, "nbformat_minor": 5 } (constraints)= # How to specify constraints ## Constraints vs bounds optimagic distinguishes between bounds and constraints. Bounds are lower and upper bounds for parameters. In the literature, they are sometimes called box constraints. Bounds are specified as `lower_bounds` and `upper_bounds` argument to `maximize` and `minimize`. Examples with bounds can be found in [this tutorial]. To specify more general constraints on your parameters, you can use the argument `constraints`. The variety of constraints you can impose ranges from rather simple ones (e.g. parameters are fixed to a value, a group of parameters is required to be equal) to more complex ones (like general linear constraints, or even nonlinear constraints). ## Can you use constraints with all optimizers? With the exception of general nonlinear constraints, we implement constraints via reparametrizations. Details are explained [here]. This means that you can use all of the constraints with any optimizer that supports bounds. Some constraints (e.g. fixing parameters) can even be used with optimizers that do not support bounds. ## Example criterion function Let's look at a variation of the sphere function to illustrate what kinds of constraints you can impose and how you specify them in optimagic: ```{eval-rst} .. code-block:: python >>> import numpy as np >>> import optimagic as om >>> def fun(params): ... offset = np.linspace(1, 0, len(params)) ... x = params - offset ... return x @ x ``` The unconstrained optimum of a six-dimensional version of this problem is: ```{eval-rst} .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([2.5, 1, 1, 1, 1, -2.5]), ... algorithm="scipy_lbfgsb", ... ) >>> res.params.round(3) # doctest: +SKIP array([1. , 0.8, 0.6, 0.4, 0.2, 0. ]) ``` The unconstrained optimum is usually easy to see because all parameters enter the criterion function in a additively separable way. ## Types of constraints Below, we show a very simple example of each type of constraint implemented in optimagic. For each constraint, we will select a subset of the parameters on which the constraint is imposed via the `selector` argument, which is a function that takes in the full parameter vector and returns the subset of parameters that should be constrained. ```{eval-rst} .. dropdown:: fixed The simplest (but very useful) constraint fixes parameters at their start values. Let's take the above example and fix the first and last parameter to 2.5 and -2.5, respectively. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([2.5, 1, 1, 1, 1, -2.5]), ... algorithm="scipy_lbfgsb", ... constraints=om.FixedConstraint( ... selector=lambda params: params[[0, 5]] ... ), ... ) Looking at the optimization result, we get: >>> res.params.round(3) array([ 2.5, 0.8, 0.6, 0.4, 0.2, -2.5]) Which is indeed the correct constrained optimum. Fixes are compatible with all optimizers. ``` ```{eval-rst} .. dropdown:: increasing In our unconstrained example, the optimal parameters are decreasing from left to right. Let's impose the constraint that the second, third and fourth parameter increase (weakly): .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([1, 1, 1, 1, 1, 1]), ... algorithm="scipy_lbfgsb", ... constraints=om.IncreasingConstraint( ... selector=lambda params: params[[1, 2, 3]] ... ), ... ) Imposing the constraint on positions ``params[[1, 2, 3]]`` means that the parameter value at index position ``2`` has to be (weakly) greater than the value at position ``1``. Likewise, the parameter value at index position ``3`` has to be (weakly) greater than the value at position ``2``. Hence, imposing an increasing constraint with only one selected parameter has no effect. We need to specify at least two parameters to make a meaningful *relative* comparison. Note that the increasing constraint affect all three parameters, i.e. ``params[1]``, ``params[2]``, and ``params[3]`` because the optimal parameters in the unconstrained case are decreasing from left to right. Looking at the optimization result, we get: >>> res.params.round(3) array([1. , 0.6, 0.6, 0.6, 0.2, 0. ]) Which is indeed the correct constrained optimum. Increasing constraints are only compatible with optimizers that support bounds. ``` ```{eval-rst} .. dropdown:: decreasing In our unconstrained example, the optimal parameters are decreasing from left to right already - without imposing any constraints. If we imposed an decreasing constraint without changing the order, it would simply have no effect. So let's impose one in a different order: .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([1, 1, 1, 1, 1, 1]), ... algorithm="scipy_lbfgsb", ... constraints=om.DecreasingConstraint( ... selector=lambda params: params[[3, 0, 4]] ... ), ... ) Imposing the constraint on positions ``params[[3, 0, 4]]`` means that the parameter value at index position ``0`` has to be (weakly) smaller than the value at position ``3``. Likewise, the parameter value at index position ``4`` has to be (weakly) smaller than the value at position ``0``. Hence, imposing a decreasing constraint with only one selected parameter has no effect. We need to specify at least two parameters to make a meaningful *relative* comparison. Note that the decreasing constraint should have no effect on ``params[4]`` because it is smaller than the other two anyways in the unconstrained optimum, but it will change the optimal values of ``params[3]`` and ``params[0]``. Indeed we get: >>> res.params.round(3) array([ 0.7, 0.8, 0.6, 0.7, 0.2, -0. ]) Which is the correct optimum. Decreasing constraints are only compatible with optimizers that support bounds. ``` ```{eval-rst} .. dropdown:: equality In our example, all optimal parameters are different. Let's constrain the first and last to be equal to each other: .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([1, 1, 1, 1, 1, 1]), ... algorithm="scipy_lbfgsb", ... constraints=om.EqualityConstraint( ... selector=lambda params: params[[0, 5]] ... ), ... ) This yields: >>> res.params.round(3) array([0.5, 0.8, 0.6, 0.4, 0.2, 0.5]) Which is the correct solution. Equality constraints are compatible with all optimizers. ``` ```{eval-rst} .. dropdown:: pairwise_equality Pairwise equality constraints are similar to equality constraints but impose that two or more groups of parameters are pairwise equal. Let's look at an example: .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([1, 1, 1, 1, 1, 1]), ... algorithm="scipy_lbfgsb", ... constraints=om.PairwiseEqualityConstraint( ... selectors=[ ... lambda params: params[[0, 1]], ... lambda params: params[[2, 3]] ... ], ... ), ... ) This constraint imposes that ``params[0] == params[2]`` and ``params[1] == params[3]``. The optimal parameters with this constraint are: >>> res.params.round(3) array([ 0.8, 0.6, 0.8, 0.6, 0.2, -0. ]) ``` ```{eval-rst} .. dropdown:: probability Let's impose the constraint that the first four parameters form valid probabilities, i.e. they should add up to one and be between zero and one. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.array([0.3, 0.2, 0.25, 0.25, 1, 1]), ... algorithm="scipy_lbfgsb", ... constraints=om.ProbabilityConstraint( ... selector=lambda params: params[:4] ... ), ... ) This yields again the correct result: .. code-block:: python >>> res.params.round(2) # doctest: +SKIP array([0.53, 0.33, 0.13, 0. , 0.2 , 0. ]) You can combine a ``ProbabilityConstraint`` with a ``FixedConstraint`` that pins some of the selected entries. The fixed values must each be in ``[0, 1)``, sum to strictly less than one, and leave at least two free entries. The remaining free entries are then optimised on the simplex that sums to ``1 - sum(fixed values)``. This is useful when part of a larger model does not need to contribute to the probability, or when some weights are set externally. ``` ```{eval-rst} .. dropdown:: covariance In many estimation problems, particularly when doing a maximum likelihood estimation, one has to estimate the covariance matrix of a random variable. The ``covariance`` costraint ensures that such a covariance matrix is always valid, i.e. positive semi-definite and symmetric. Due to its symmetry, only the lower triangle of a covariance matrix actually has to be estimated. Let's look at an example. We want to impose that the first three elements form the lower triangle of a valid covariance matrix. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.ones(6), ... algorithm="scipy_lbfgsb", ... constraints=om.FlatCovConstraint( ... selector=lambda params: params[:3] ... ), ... ) This yields the same solution as an unconstrained estimation because the constraint is not binding: >>> res.params.round(3) array([ 1.006, 0.784, 0.61 , 0.4 , 0.2 , -0. ]) We can now use one of optimagic's utility functions to actually build the covariance matrix out of the first three parameters: .. code-block:: python >>> from optimagic.utilities import cov_params_to_matrix >>> cov_params_to_matrix(res.params[:3]).round(2) # doctest: +NORMALIZE_WHITESPACE array([[1.01, 0.78], [0.78, 0.61]]) ``` ```{eval-rst} .. dropdown:: sdcorr ``sdcorr`` constraints are very similar to ``covariance`` constraints. The only difference is that instead of estimating a covariance matrix, we estimate standard deviations and the correlation matrix of random variables. Let's look at an example. We want to impose that the first three elements form valid standard deviations and a correlation matrix. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.ones(6), ... algorithm="scipy_lbfgsb", ... constraints=om.FlatSDCorrConstraint( ... selector=lambda params: params[:3] ... ), ... ) This yields the same solution as an unconstrained estimation because the constraint is not binding: >>> res.params.round(3) # doctest: +SKIP array([ 1. , 0.8, 0.6, 0.4, 0.2, -0. ]) We can now use one of optimagic's utility functions to actually build the standard deviations and the correlation matrix: .. code-block:: python >>> from optimagic.utilities import sdcorr_params_to_sds_and_corr >>> sd, corr = sdcorr_params_to_sds_and_corr(res.params[:3]) >>> sd.round(2) array([1. , 0.8]) >>> corr.round(2) # doctest: +NORMALIZE_WHITESPACE array([[1. , 0.6], [0.6, 1. ]]) ``` ```{eval-rst} .. dropdown:: linear Linear constraints are the most difficult but also the most powerful constraints in your toolkit. They can be used to express constraints of the form ``lower_bound <= weights.dot(x) <= upper_bound`` or ``weights.dot(x) = value`` where ``x`` are the selected parameters. Linear constraints have many of the other constraint types as special cases, but typically it is more convenient to use the special cases instead of expressing them as a linear constraint. Internally, it will make no difference. Let's impose the constraint that the average of the first four parameters is at least 0.95. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.ones(6), ... algorithm="scipy_lbfgsb", ... constraints=om.LinearConstraint( ... selector=lambda params: params[:4], ... lower_bound=0.95, ... weights=0.25, ... ), ... ) This yields: >>> res.params.round(2) array([ 1.25, 1.05, 0.85, 0.65, 0.2 , -0. ]) Where the first four parameters have an average of 0.95. In the above example, ``lower_bound`` and ``weights`` are scalars. They may, however, also be arrays (or even pytrees) with bounds and weights for each selected parameter. ``` ```{eval-rst} .. dropdown:: nonlinear .. warning:: General nonlinear constraints that are specified via a black-box constraint function can only be used if you choose an optimizer that supports it. This feature is currently supported by the algorithms: * ``ipopt`` * ``nlopt``: ``cobyla``, ``slsqp``, ``isres``, ``mma`` * ``scipy``: ``cobyla``, ``slsqp``, ``trust_constr`` You can use nonlinear constraints to express restrictions of the form ``lower_bound <= func(x) <= upper_bound`` or ``func(x) = value`` where ``x`` are the selected parameters and ``func`` is the constraint function. Let's impose the constraint that the product of all but the last parameter is 1. .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.ones(6), ... algorithm="scipy_slsqp", ... constraints=om.NonlinearConstraint( ... selector=lambda params: params[:-1], ... func=lambda x: np.prod(x), ... value=1.0, ... ), ... ) This yields: >>> res.params.round(2) array([ 1.31, 1.16, 1.01, 0.87, 0.75, -0. ]) Where the product of all but the last parameters is equal to 1. If you have a function that calculates the derivative of your constraint, you can add this under the key `"derivative"` to the constraint dictionary. Otherwise, numerical derivatives are calculated for you if needed. ``` ## Imposing multiple constraints at once The above examples all just impose one constraint at a time. To impose multiple constraints simultaneously, simple pass in a list of constraints. For example: ```{eval-rst} .. code-block:: python >>> res = om.minimize( ... fun=fun, ... params=np.ones(6), ... algorithm="scipy_lbfgsb", ... constraints=[ ... om.EqualityConstraint(selector=lambda params: params[:2]), ... om.LinearConstraint( ... selector=lambda params: params[2:5], ... weights=1, ... value=3, ... ), ... ], ... ) This yields: >>> res.params.round(2) array([0.9, 0.9, 1.2, 1. , 0.8, 0. ]) There are limits regarding the compatibility of overlapping constraints. You will get a descriptive error message if your constraints are not compatible. ``` ## How to select the parameters? The parameters can be selected via a `selector` function. This function takes in the full parameter vector and returns the subset of parameters that should be constrained. Let's assume we have defined parameters in a nested dictionary: ```python params = {"a": np.ones(2), "b": {"c": 3, "d": pd.Series([4, 5])}} ``` It is probably not a good idea to use a nested dictionary for so few parameters, but let's ignore that. Now assume we want to fix the parameters in the pandas Series at their start values. We can do so as follows: ```python res = om.minimize( fun=some_fun, params=params, algorithm="scipy_lbfgsb", constraints=om.FixedConstraint(selector=lambda params: params["b"]["d"]), ) ``` I.e. the value corresponding to `selector` is a python function that takes the full `params` and returns a subset. The selected subset does not have to be a numpy array, it can be an arbitrary pytree. Using lambda functions if often convenient, but we could have just as well defined the selector function using def. ```python def my_selector(params): return params["b"]["d"] res = om.minimize( fun=some_fun, params=params, algorithm="scipy_lbfgsb", constraints=om.FixedConstraint(selector=my_selector), ) ``` [here]: ../../explanation/implementation_of_constraints.md [this tutorial]: ../tutorials/optimization_overview.ipynb { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(how-to-jac)=\n", "\n", "# How to speed up your optimization using derivatives\n", "\n", "Many optimization algorithms use derivatives to find good search directions. If you \n", "use a derivative based optimizer but do not provide derivatives of your objective \n", "function, optimagic calculates a numerical derivative for you. \n", "\n", "While this numerical derivative is usually precise enough to find good search directions \n", "it requires `n + 1` evaluations of the objective function (where `n` is the number of \n", "free parameters). For large `n` this becomes very slow.\n", "\n", "This how-to guide shows how you can speed up your optimization by parallelizing \n", "numerical derivatives or by providing closed form derivatives. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parallel numerical derivatives\n", "\n", "If you have a computer with a few idle cores, the easiest way to speed up your\n", "optimization with a gradient based optimizer is to calculate numerical derivatives \n", "in parallel:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om\n", "\n", "\n", "def sphere(x):\n", " return x @ x\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " numdiff_options=om.NumdiffOptions(n_cores=6),\n", ")\n", "res.params.round(6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Of course, for this super fast objective function, parallelizing will not yield an actual \n", "speedup. But if your objective function takes 100 milliseconds or longer to evaluate, \n", "you can parallelize efficiently to up to `n + 1` cores. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Custom derivatives\n", "\n", "If you don't want to solve your speed problem by throwing more compute at it, you can \n", "provide a derivative to optimagic that is faster than doing `n + 1` evaluations of `fun`. \n", "Here we show you how to hand-code it, but in practice you would usually use JAX or another \n", "autodiff framework to create the derivative." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere_gradient(x):\n", " return 2 * x\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " jac=sphere_gradient,\n", ")\n", "res.params.round(6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this example, the evaluation of `sphere_gradient` is even faster than evaluating `sphere`. \n", "\n", "In non-trivial functions, there are synergies between calculating the objective value and \n", "its derivative. Therefore, you can also provide a function that evaluates both at the same time. In such a case, providing fun is optional." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere_fun_and_gradient(x):\n", " return x @ x, 2 * x\n", "\n", "\n", "res = om.minimize(\n", " fun=sphere, # optional when fun_and_jac is provided\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " fun_and_jac=sphere_fun_and_gradient,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "`fun_and_jac` can be provided in addition to or instead of `jac` or `fun`. Providing them \n", "together gives optimagic more opportunities to save \n", "time by evaluating just the function that is needed for a given optimizer. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Derivatives with flexible params\n", "\n", "Derivatives are compatible with any format of params. In general, the gradients have \n", "just the same structure as your params. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def dict_fun(x):\n", " return x[\"a\"] ** 2 + x[\"b\"] ** 4\n", "\n", "\n", "def dict_gradient(x):\n", " return {\"a\": 2 * x[\"a\"], \"b\": 4 * x[\"b\"] ** 3}\n", "\n", "\n", "res = om.minimize(\n", " fun=dict_fun,\n", " params={\"a\": 1, \"b\": 2},\n", " algorithm=\"scipy_lbfgsb\",\n", " jac=dict_gradient,\n", ")\n", "res.params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is also the convention that JAX uses, so any derivative you get via JAX will be \n", "compatible with optimagic. \n", "\n", "## Derivatives for least-squares functions\n", "\n", "When minimizing least-squares functions, you don't need the gradient of the objective \n", "value but the jacobian of the least-squares residuals. Moreover, this jacobian function \n", "needs to be decorated with the `mark.least_squares` decorator. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "@om.mark.least_squares\n", "def ls_sphere(params):\n", " return params\n", "\n", "\n", "@om.mark.least_squares\n", "def ls_sphere_jac(params):\n", " return np.eye(len(params))\n", "\n", "\n", "res = om.minimize(\n", " fun=ls_sphere,\n", " params=np.arange(3),\n", " algorithm=\"scipy_ls_lm\",\n", " jac=ls_sphere_jac,\n", ")\n", "res.params.round(5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `fun_and_jac` argument works just analogous to the scalar case. \n", "\n", "Derivatives of least-squares functions again work with all valid formats of params. \n", "However, the structure of the jacobian can be a bit complicated. Again, JAX will do \n", "the right thing here, so we strongly suggest you calculate all your jacobians via JAX,\n", "especially if your params are not a flat numpy array. \n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Derivatives that work for scalar and least-squares optimizers\n", "\n", "If you want to seamlessly switch between scalar and least-squares optimizers, you can \n", "do so by providing even more versions of derivatives to `minimize`. You probably won't \n", "ever need this, but here is how you would do it. To pretend that this can be useful, \n", "we compare a scalar and a least squares optimizer in a criterion_plot:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results = {}\n", "for algorithm in [\"scipy_lbfgsb\", \"scipy_ls_lm\"]:\n", " results[algorithm] = om.minimize(\n", " fun=ls_sphere,\n", " params=np.arange(5),\n", " algorithm=algorithm,\n", " jac=[sphere_gradient, ls_sphere_jac],\n", " )\n", "\n", "fig = om.criterion_plot(results)\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that both optimizers were super fast in solving this problem (mainly because the problem is so simple) and in this case the scalar optimizer was even faster. However, in non-trivial problems it almost always pays of to exploit the least-squares structure if you can." ] } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 2 } # How to document optimizers This guide shows you how to document algorithms in optimagic using our new documentation system. We'll walk through the process step-by-step using the `ScipyLBFGSB` optimizer as a complete example. ## When to Use This Guide Use this guide when you need to: - Document a new algorithm you've added to optimagic - Migrate existing algorithm documentation from the old split system (docstrings + `algorithms.md`) to the new system - Update or improve existing algorithm documentation If you're adding a completely new optimizer to optimagic, start with the "How to Add Optimizers guide" first, then use this guide to document your algorithm properly. ## Why the New Documentation System? Previously, algorithm documentation was scattered across multiple places: - Basic descriptions in the algorithm class docstrings - Detailed parameter descriptions in `algorithms.md` - Usage examples separate from the algorithm definitions This made it hard to maintain consistency and keep documentation up-to-date. The new system centralizes nearly all documentation in the algorithm code itself, making it: - Easier to maintain (documentation lives next to code) - More consistent (unified format across all algorithms) - Auto-generated (parameter lists appear automatically in docs) - Type-safe (documentation matches actual parameter types) ## The Documentation System Components Our documentation system has three main parts: 1. **Algorithm Class Documentation**: A comprehensive docstring in the algorithm dataclass that explains what the algorithm does, how it works, and when to use it 1. **Parameter Documentation**: Detailed docstrings for each parameter with mathematical formulations when needed 1. **Usage Integration**: A section in `algorithms.md` that show how to use the algorithm Let's walk through documenting an algorithm from start to finish. ## Example: Documenting ScipyLBFGSB We'll use the `ScipyLBFGSB` optimizer to show you exactly how to document an algorithm. This is a real example from the optimagic codebase, so you can follow along and see the results. ### Step 1: Understand Your Algorithm Before writing documentation, make sure you understand: - What the algorithm does mathematically - What problems it's designed to solve - How its parameters affect behavior - Any performance characteristics or limitations For L-BFGS-B, this means understanding it's a quasi-Newton method for bound-constrained optimization that approximates the Hessian using gradient history. ```{eval-rst} .. note:: If you are simply migrating an existing algorithm, you can mostly rely on the existing documentation in the algorithm class docstring and `algorithms.md`. ``` ### Step 2: Write the Algorithm Class Documentation The algorithm class docstring is the most important part. It should give users everything they need to decide whether to use this algorithm. Here's how we document `ScipyLBFGSB`: ```python # src/optimagic/optimizers/scipy_optimizers.py class ScipyLBFGSB(Algorithm): """Minimize a scalar differentiable function using the L-BFGS-B algorithm. The optimizer is taken from scipy, which calls the Fortran code written by the original authors of the algorithm. The Fortran code includes the corrections and improvements that were introduced in a follow up paper. lbfgsb is a limited memory version of the original bfgs algorithm, that deals with lower and upper bounds via an active set approach. The lbfgsb algorithm is well suited for differentiable scalar optimization problems with up to several hundred parameters. It is a quasi-newton line search algorithm. At each trial point it evaluates the criterion function and its gradient to find a search direction. It then approximates the hessian using the stored history of gradients and uses the hessian to calculate a candidate step size. Then it uses a gradient based line search algorithm to determine the actual step length. Since the algorithm always evaluates the gradient and criterion function jointly, the user should provide a ``fun_and_jac`` function that exploits the synergies in the calculation of criterion and gradient. The lbfgsb algorithm is almost perfectly scale invariant. Thus, it is not necessary to scale the parameters. """ ``` **What makes this docstring effective:** - **Clear first line**: States exactly what the algorithm does - **Implementation details**: Explains it uses scipy's Fortran implementation - **Algorithm classification**: Identifies it as a quasi-Newton method - **Problem suitability**: Explains what problems it's good for - **How it works**: Brief explanation of the algorithm's approach - **Performance characteristics**: Mentions scale invariance - **Usage advice**: Suggests using `fun_and_jac` for efficiency ### Step 3: Document Individual Parameters Each parameter needs clear documentation explaining what it controls and how it affects the algorithm's behavior. ```python # Basic parameter documentation stopping_maxiter: PositiveInt = STOPPING_MAXITER """Maximum number of iterations.""" # Parameter with mathematical formulation convergence_ftol_rel: NonNegativeFloat = CONVERGENCE_FTOL_REL r"""Converge if the relative change in the objective function is less than this value. More formally, this is expressed as. .. math:: \frac{f^k - f^{k+1}}{\max\{|f^k|, |f^{k+1}|, 1\}} \leq \textsf{convergence_ftol_rel}. """ # Parameter with external library context limited_memory_storage_length: PositiveInt = LIMITED_MEMORY_STORAGE_LENGTH """The maximum number of variable metric corrections used to define the limited memory matrix. This is the 'maxcor' parameter in the SciPy documentation. The default value is taken from SciPy's L-BFGS-B implementation. Larger values use more memory but may converge faster for some problems. """ ``` **Key principles for parameter documentation:** - **Start with a clear description** of what the parameter controls - **Add mathematical formulations** when they clarify the exact meaning (use `r"""` for raw strings with LaTeX) - **Include external library context** when relevant (e.g., "Default value is taken from SciPy") - **Explain performance implications** when they matter - **Use proper type annotations** that match the parameter's constraints ```{eval-rst} .. warning:: If your optimizer module uses type hints (e.g., ``PositiveInt``, ``NonNegativeInt``), include the following at the top of your optimizer module: .. code-block:: python from __future__ import annotations Without this, type hints such as ``PositiveInt`` may appear decomposed in the documentation (e.g., as ``Annotated[int, Gt(gt=0)]``). ``` ### Step 4: Integrate into `algorithms.md` The final step is integrating your documented algorithm into the main documentation. This creates a dropdown section that shows users how to use the algorithm. Add the following to `docs/source/algorithms.md` in an `eval-rst` block: ```text .. dropdown:: scipy_lbfgsb **How to use this algorithm:** .. code-block:: python import optimagic as om om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.scipy_lbfgsb(stopping_maxiter=1_000, ...), ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="scipy_lbfgsb", algo_options={"stopping_maxiter": 1_000, ...}, ) **Description and available options:** .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyLBFGSB ``` **What this section provides:** - **The dropdown button and title**: Makes it easy to find the algorithm - **Concrete usage examples** showing both the object and string interfaces - **Algorithm-specific parameter** in the usage example - **Auto-generated documentation** via the `autoclass` directive that pulls in your docstrings ## Working with Existing Documentation If you're migrating an algorithm that already has documentation: ### Finding Existing Content Look for existing documentation in: - **Algorithm class docstrings**: Usually basic descriptions - **`docs/source/algorithms.md`**: Detailed parameter descriptions and examples - **Research papers**: For mathematical formulations and background - **External library docs**: For default values and parameter meanings ### Migration Strategy 1. **Start with the algorithm class**: Move the best description from `algorithms.md` to the class docstring 1. **Update and expand**: Add missing information about performance, usage, etc. 1. **Move parameter docs**: Transfer parameter descriptions from `algorithms.md` to individual parameter docstrings 1. **Verify accuracy**: Check that all information is current and correct 1. **Create new integration**: Replace the old `algorithms.md` section with the new dropdown format ## Common Pitfalls to Avoid - **Don't copy-paste generic descriptions**: Each algorithm needs specific, detailed documentation - **Don't skip mathematical formulations**: When convergence criteria or parameters have precise mathematical definitions, include them - **Don't ignore external library context**: Always mention where default values come from - **Don't use vague parameter descriptions**: "Controls the algorithm behavior" is not helpful - **Don't forget performance implications**: Users need to understand trade-offs between parameters ## Getting Help If you're stuck or need clarification: - Look at existing well-documented algorithms like `ScipyLBFGSB` - Check the {ref}`style_guide` for coding conventions - Ask questions in GitHub issues or discussions The goal is to make optimagic's algorithm documentation the best resource for understanding and using optimization algorithms effectively. { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "(how-to-errors)=\n", "\n", "# How to handle errors during optimization\n", "\n", "## Try to avoid errors\n", "\n", "Often, optimizers try quite extreme parameter vectors, which then can raise errors in your criterion function or derivative. Often, there are simple tricks to make your code more robust. Avoiding errors is always better than dealing with errors after they occur. \n", "\n", "- Avoid to take ``np.exp`` without further safeguards. With 64 bit floating point numbers, the exponential function is only well defined roughly between -700 and 700. Below it is 0, above it is inf. Sometimes you can use ``scipy.special.logsumexp`` to avoid unsafe evaluations of the exponential. Read [this](https://en.wikipedia.org/wiki/LogSumExp) for background information on the logsumexp trick.\n", "- Set bounds for your parameters that prevent extreme parameter constellations.\n", "- Use the ``bounds_distance`` option with a not too small value for ``covariance`` and ``sdcorr`` constraints.\n", "- Use `optimagic.utilities.robust_cholesky` instead of normal\n", " cholesky decompositions or try to avoid cholesky decompositions.\n", "- Use a less aggressive optimizer. Trust region optimizers like `fides` usually choose less extreme steps in the beginnig than line search optimizers like `scipy_bfgs` and `scip_lbfgsb`. \n", "\n", "## Do not use clipping\n", "\n", "A commonly chosen solution to numerical problems is clipping of extreme values. Naive clipping leads to flat areas in your criterion function and can cause spurious convergence. Only use clipping if you know that your optimizer can deal with flat parts. " ] }, { "cell_type": "markdown", "id": "1", "metadata": {}, "source": [ "## Let optimagic do its magic\n", "\n", "Instead of avoiding errors in your criterion function, you can raise them and let optimagic deal with them. If you are using numerical derivatives, errors will automatically be raised if any entry in the derivative is not finite. \n", "\n", "### An example\n", "\n", "Let's look at a simple example from the Moré-Wild benchmark set that has a numerical instability. " ] }, { "cell_type": "code", "execution_count": null, "id": "2", "metadata": {}, "outputs": [], "source": [ "import warnings\n", "\n", "import numpy as np\n", "import plotly.io as pio\n", "from scipy.optimize import minimize as scipy_minimize\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om\n", "\n", "warnings.simplefilter(\"ignore\")" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "def jennrich_sampson(x):\n", " dim_out = 10\n", " fvec = (\n", " 2 * (1.0 + np.arange(1, dim_out + 1))\n", " - np.exp(np.arange(1, dim_out + 1) * x[0])\n", " - np.exp(np.arange(1, dim_out + 1) * x[1])\n", " )\n", " return fvec @ fvec\n", "\n", "\n", "correct_params = np.array([0.2578252135686162, 0.2578252135686162])\n", "correct_criterion = 124.3621823556148\n", "\n", "start_x = np.array([0.3, 0.4])" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "### What would scipy do?" ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "scipy_res = scipy_minimize(jennrich_sampson, x0=start_x, method=\"L-BFGS-B\")" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "scipy_res.success" ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [], "source": [ "correct_params.round(4), scipy_res.x.round(4)" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "So, scipy thinks it solved the problem successfully but the result is far off. (Note that scipy would have given us a warning, but we disabled warnings in order to not clutter the output).\n", "\n", "### optimagic's error handling magic" ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=jennrich_sampson,\n", " params=start_x,\n", " algorithm=\"scipy_lbfgsb\",\n", " error_handling=\"continue\",\n", ")\n", "\n", "correct_params, res.params" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "### How does the magic work\n", "\n", "When an error occurs and `error_handling` is set to `\"continue\"`, optimagic replaces your criterion with a dummy function (and adjusts the derivative accordingly). \n", "\n", "The dummy function has two important properties:\n", "\n", "1. Its value is always higher than criterion at start params. \n", "2. Its slope guides the optimizer back towards the start parameters. I.e., if you are minimizing, the direction of strongest decrease is towards the start parameters; if you are maximizing, the direction of strongest increase is towards the start parameters. \n", "\n", "Therefore, when hitting an undefined area, an optimizer can take a few steps back until it is in better territory and then continue its work. \n", "\n", "Importantly, the optimizer will not simply go back to a previously evaluated point (which would just lead to cyclical behavior). It will just go back in the direction it originally came from.\n", "\n", "In the concrete example, the dummy function would look similar to the following:" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "def dummy(params):\n", " start_params = np.array([0.3, 0.4])\n", " # this is close to the actual value used by optimagic\n", " constant = 8000\n", " # the actual slope used by optimagic would be even smaller\n", " slope = 10_000\n", " diff = params - start_params\n", " return constant + slope * np.linalg.norm(diff)" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "Now, let's plot the two functions. For better illustration, we assume that the jennrich_sampson function is only defined until it reaches a value of 100_000 and the dummy function takes over from there. " ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "from plotly import graph_objects as go\n", "\n", "grid = np.linspace(0, 1)\n", "params = [np.full(2, val) for val in grid]\n", "values = np.array([jennrich_sampson(p) for p in params])\n", "values = np.where(values <= 1e5, values, np.nan)\n", "dummy_values = np.array([dummy(p) for p in params])\n", "dummy_values = np.where(np.isfinite(values), np.nan, dummy_values)" ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "fig = go.Figure()\n", "fig.add_trace(go.Scatter(x=grid, y=values))\n", "fig.add_trace(go.Scatter(x=grid, y=dummy_values))\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "We can see that the dummy function is lower than the highest achieved value of `jennrich_sampson` but higher than the start values. It is also rather flat. Fortunately, that is all we need. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 5 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to choose a strategy for global optimization\n", "\n", "(to be written)" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 2 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "(how-to-logging)=\n", "\n", "# How to use logging\n", "\n", "\n", "optimagic can keep a persistent log of the parameter and criterion values tried out by an optimizer in a sqlite database. \n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Turn logging on or off\n", "\n", "To enable logging, it suffices to provide a path to an sqlite database when calling ``maximize`` or ``minimize``. The database does not have to exist, optimagic will generate it for you. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "import numpy as np\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere(params):\n", " return params @ params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Remove the log file if it exists (just needed for the example)\n", "log_file = Path(\"my_log.db\")\n", "if log_file.exists():\n", " log_file.unlink()\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " logging=\"my_log.db\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In case the SQLite file already exists, this will raise a `FileExistsError` to prevent from accidentally polluting an existing database. If you want to reuse\n", "an existing database on purpose, you must explicitly provide the corresponding option for `if_database_exists`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "log_options = om.SQLiteLogOptions(\n", " \"my_log.db\", if_database_exists=om.ExistenceStrategy.EXTEND\n", ")\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " logging=log_options,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Make logging faster\n", "\n", "By default, we use a very safe mode of sqlite that makes it almost impossible to corrupt the database. Even if your computer is suddenly shut down or unplugged. \n", "\n", "However, this makes writing logs rather slow, which becomes notable when the criterion function is very fast. \n", "\n", "In that case, you can enable `fast_logging`, which is still quite safe!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "log_options = om.SQLiteLogOptions(\n", " \"my_log.db\",\n", " fast_logging=True,\n", " if_database_exists=om.ExistenceStrategy.REPLACE,\n", ")\n", "\n", "res = om.minimize(\n", " fun=sphere,\n", " params=np.arange(5),\n", " algorithm=\"scipy_lbfgsb\",\n", " logging=log_options,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Reading the log\n", "To read the log after an optimization, extract the logger from the optimization result:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader = res.logger" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Alternatively, you can create the reader like this:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader = om.SQLiteLogReader(\"my_log.db\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read the start params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader.read_start_params()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read a specific iteration (use -1 for the last)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader.read_iteration(-1)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read the full history" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "reader.read_history().keys()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plot the history from a log" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(\"my_log.db\")\n", "fig.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.params_plot(\"my_log.db\", selector=lambda x: x[1:3])\n", "fig.show()" ] } ], "metadata": { "interpreter": { "hash": "5cdb9867252288f10687117449de6ad870b49795ca695c868016dc0022895cce" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 2 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "(how-to-multistart)=\n", "\n", "# How to do multistart optimizations\n", "\n", "Sometimes you want to make sure that your optimization is robust to the initial\n", "parameter values, i.e. that it does not get stuck at a local optimum. This is where\n", "multistart comes in handy.\n", "\n", "\n", "## What does multistart (not) do\n", "\n", "In short, multistart iteratively runs local optimizations from different initial\n", "conditions. If enough local optimization convergence to the same point, it stops.\n", "Importantly, it cannot guarantee that the result is the global optimum, but it can\n", "increase your confidence in the result.\n", "\n", "## TL;DR\n", "\n", "To activate multistart at the default options, pass `multistart=True` to the `minimize`\n", "or `maximize` function, as well as finite bounds on the parameters (which are used to\n", "sample the initial points). The default options are discussed below." ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om\n", "\n", "\n", "def fun(x):\n", " return x @ x\n", "\n", "\n", "x0 = np.arange(7) - 4\n", "\n", "bounds = om.Bounds(\n", " lower=np.full_like(x0, -5),\n", " upper=np.full_like(x0, 10),\n", ")\n", "\n", "algo_options = {\"stopping_maxfun\": 1_000}\n", "\n", "res = om.minimize(\n", " fun=fun,\n", " x0=x0,\n", " algorithm=\"scipy_neldermead\",\n", " algo_options=algo_options,\n", " bounds=bounds,\n", " multistart=True,\n", ")" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "In this example, we limited each local optimization to 1_000 function evaluations. In\n", "general, it is a good idea to limit the number of iterations and function evaluations\n", "for the local optimization. Because of the iterative nature of multistart, this\n", "limitation will usually not result in a precision issue." ] }, { "cell_type": "markdown", "id": "3", "metadata": {}, "source": [ "## What does multistart mean in optimagic?\n", "\n", "Our multistart optimizations are inspired by the [TikTak algorithm](https://github.com/serdarozkan/TikTak) and consist of the following steps:\n", "\n", "1. Draw a large exploration sample of parameter vectors randomly or using a\n", " low-discrepancy sequence.\n", "1. Evaluate the objective function in parallel on the exploration sample.\n", "1. Sort the parameter vectors from best to worst according to their objective function\n", " values. \n", "1. Run local optimizations iteratively. That is, the first local optimization is started\n", " from the best parameter vector in the sample. All subsequent ones are started from a\n", " convex combination of the currently best known parameter vector and the next sample\n", " point. " ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## Visualizing multistart results\n", "\n", "To illustrate the multistart results, we will consider the optimization of a slightly\n", "more complex objective function, compared to `fun` from above. We also limit the\n", "number of exploration samples to 100." ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "def alpine(x):\n", " return np.sum(np.abs(x * np.sin(x) + 0.1 * x))\n", "\n", "\n", "res = om.minimize(\n", " alpine,\n", " x0=x0,\n", " algorithm=\"scipy_neldermead\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(n_samples=100, seed=0),\n", ")\n", "\n", "fig = om.criterion_plot(res, monotone=True)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ "In the above image we see the optimization history for all of the local optimizations\n", "that have been run by multistart. The turquoise line represents the history\n", "corresponding to the local optimization that found the overall best parameter.\n", "\n", "We see that running a single optimization would not have sufficed, as some local\n", "optimizations are stuck." ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "## Multistart does not always run many optimization\n", "\n", "Since the local optimizations are run iteratively by multistart, it is possible that\n", "only a handful of optimizations are actually run if all of them converge to the same\n", "point. This convergence is determined by the `convergence_max_discoveries` option,\n", "which defaults to 2. This means that if 2 local optimizations report the same point,\n", "multistart will stop. Below we see that if we use the simpler objective function\n", "(`fun`), and the `scipy_lbfgsb` algorithm, multistart runs only 2 local optimizations,\n", "and then stops, as both of them converge to the same point. Note that, the\n", "`scipy_lbfgsb` algorithm can solve this simple problem precisely, without reaching the\n", "maximum number of function evaluations." ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun,\n", " x0=x0,\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(n_samples=100, seed=0),\n", ")\n", "\n", "fig = om.criterion_plot(res)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## How to configure multistart\n", "\n", "Configuration of multistart can be done by passing an instance of\n", "`optimagic.MultistartOptions` to `minimize` or `maximize`. Let's look at a few examples\n", "configurations." ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "### How to run a specific number of optimizations\n", "\n", "To run a specific number of local optimizations, you need to set the `stopping_maxopt`\n", "option. Note that this does not set the number of exploration samples, which is\n", "controlled by the `n_samples` option. The number of exploration samples always needs\n", "to be at least as large as the number of local optimizations.\n", "\n", "Note that, as long as `convergence_max_discoveries` is smaller than `stopping_maxopt`,\n", "it is possible that a smaller number of local optimizations are run. To avoid this,\n", "set `convergence_max_discoveries` to a value at least as large as `stopping_maxopt`.\n", "\n", "To run, for example, 10 local optimizations from 15 exploration samples, do:" ] }, { "cell_type": "code", "execution_count": null, "id": "11", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " alpine,\n", " x0=x0,\n", " algorithm=\"scipy_neldermead\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(\n", " n_samples=15,\n", " stopping_maxopt=10,\n", " convergence_max_discoveries=10,\n", " ),\n", ")\n", "\n", "res.multistart_info.n_optimizations" ] }, { "cell_type": "markdown", "id": "12", "metadata": {}, "source": [ "### How to set a custom exploration sample\n", "\n", "If you want to start the multistart algorithm with a custom exploration sample, you can\n", "do so by passing a sequence of parameters to the `sample` option. Note that sequence\n", "elements must be of the same type as your parameter.\n", "\n", "To generate a sample of 100 random parameters and run them through the multistart\n", "algorithm, do:" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(12345)\n", "\n", "sample = [x0 + rng.uniform(-1, 1, size=len(x0)) for _ in range(100)]\n", "\n", "res = om.minimize(\n", " alpine,\n", " x0=x0,\n", " algorithm=\"scipy_neldermead\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(sample=sample),\n", ")" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "### How to run multistart in parallel\n", "\n", "\n", "The multistart algorithm can be run in parallel by setting the `n_cores` option to a\n", "value greater than 1. This will run the algorithm in batches. By default, the batch\n", "size is set to `n_cores`, but can be controlled by setting the `batch_size` option. The\n", "default batch evaluator is `joblib`, but can be controlled by setting the\n", "`batch_evaluator` option to `\"pathos\"` or a custom callable.\n", "\n", "To run the multistart algorithm in parallel, do:" ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " alpine,\n", " x0=x0,\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(n_cores=2),\n", ")" ] }, { "cell_type": "markdown", "id": "16", "metadata": {}, "source": [ "## What to do if you do not have bounds\n", "\n", "Multistart requires finite bounds on the parameters. If your optimization problem is not\n", "bounded, you can set soft lower and upper bounds. These bounds will only be used to\n", "draw the exploration sample, and will not be used to constrain the local optimizations.\n", "\n", "To set soft bounds, do:" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " alpine,\n", " x0=x0,\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=om.Bounds(soft_lower=np.full_like(x0, -3), soft_upper=np.full_like(x0, 8)),\n", " multistart=True,\n", ")" ] }, { "cell_type": "markdown", "id": "18", "metadata": {}, "source": [ "## Understanding multistart results\n", "\n", "When activating multistart, the optimization result object corresponds to the local\n", "optimization that found the best objective function value. The result object has the\n", "additional attribute `multistart_info`, where all of the additional information is\n", "stored. It has the following attributes:\n", "\n", "- `local_optima`: A list with the results from all local optimizations that were performed.\n", "- `start_parameters`: A list with the start parameters from those optimizations \n", "- `exploration_sample`: A list with parameter vectors at which the objective function was evaluated in an initial exploration phase. \n", "- `exploration_results`: The corresponding objective values.\n", "- `n_optimizations`: The number of local optimizations that were run.\n", "\n", "To illustrate the multistart results, let us consider the optimization of the simple\n", "`fun` objective function from above." ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun,\n", " x0=x0,\n", " algorithm=\"scipy_lbfgsb\",\n", " bounds=bounds,\n", " algo_options=algo_options,\n", " multistart=om.MultistartOptions(n_samples=100, convergence_max_discoveries=2),\n", ")" ] }, { "cell_type": "markdown", "id": "20", "metadata": {}, "source": [ "### Start parameters\n", "\n", "The start parameters are the parameter vectors from which the local optimizations were\n", "started. Since the default number of `convergence_max_discoveries` is 2, and both\n", "local optimizations were successfull, the start parameters have 2 rows." ] }, { "cell_type": "code", "execution_count": null, "id": "21", "metadata": {}, "outputs": [], "source": [ "res.multistart_info.start_parameters" ] }, { "cell_type": "markdown", "id": "22", "metadata": {}, "source": [ "### Local Optima\n", "\n", "The local optima are the results from the local optimizations. Since in this example\n", "only two local optimizations were run, the local optima list has two elements, each of\n", "which is an optimization result object." ] }, { "cell_type": "code", "execution_count": null, "id": "23", "metadata": {}, "outputs": [], "source": [ "len(res.multistart_info.local_optima)" ] }, { "cell_type": "markdown", "id": "24", "metadata": {}, "source": [ "### Exploration sample\n", "\n", "The exploration sample is a list of parameter vectors at which the objective function\n", "was evaluated. Above, we chose a random exploration sample of 100 parameter vectors." ] }, { "cell_type": "code", "execution_count": null, "id": "25", "metadata": {}, "outputs": [], "source": [ "np.vstack(res.multistart_info.exploration_sample).shape" ] }, { "cell_type": "markdown", "id": "26", "metadata": {}, "source": [ "### Exploration results\n", "\n", "The exploration results are the objective function values at the exploration sample." ] }, { "cell_type": "code", "execution_count": null, "id": "27", "metadata": {}, "outputs": [], "source": [ "len(res.multistart_info.exploration_results)" ] }, { "cell_type": "markdown", "id": "28", "metadata": {}, "source": [ "### Number of local optimizations" ] }, { "cell_type": "code", "execution_count": null, "id": "29", "metadata": {}, "outputs": [], "source": [ "res.multistart_info.n_optimizations" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 5 } (scaling)= # How to scale optimization problems Real world optimization problems often comprise parameters of vastly different orders of magnitudes. This is typically not a problem for gradient based optimization algorithms but can considerably slow down derivative free optimizers. Below we describe three simple heuristics to improve the scaling of optimization problems and discuss the pros and cons of each approach. ## What does well scaled mean In short, an optimization problem is well scaled if a fixed step in any direction yields a roughly similar sized change in the objective function. In practice, this can never be achieved perfectly (at least for nonlinear problems). However, one can easily improve over simply ignoring the problem altogether. ## TL;DR To activate scaling at the default options, pass `scaling=True` to the `minimize` or `maximize` function. This uses the start values heuristic explained below. The default options are discussed in the section {ref}`scaling-default-values`. ```{code-block} python --- emphasize-lines: 13 --- import numpy as np import optimagic as om def fun(x): return x @ x res = om.minimize( fun=fun, x0=np.arange(5), algorithm="scipy_lbfgsb", scaling=True, ) ``` ## Heuristics to improve scaling (scaling-start-values-heuristic)= ### Divide by absolute value of start parameters In many applications, parameters with very large start values will vary over a wide range and a change in that parameter will only lead to a relatively small change in the objective function. If this is the case, the scaling of the optimization problem can be improved by simply dividing all parameter vectors by the start parameters. **Advantages:** - Straightforward - Works with any type of constraints **Disadvantages:** - Makes scaling dependent on start values - Parameters with zero start value need special treatment **How to specify this scaling:** ```{code-block} python --- emphasize-lines: 5 --- res = om.minimize( fun=fun, x0=np.arange(5), algorithm="scipy_lbfgsb", scaling=om.ScalingOptions(method="start_values", clipping_value=0.1), ) ``` ### Divide by bounds In many optimization problems, one has additional information on bounds of the parameter space. Some of these bounds are hard (e.g. probabilities or variances are non negative), others are soft and derived from simple considerations (e.g. if a time discount factor were smaller than 0.7, we would not observe anyone to pursue a university degree in a structural model of educational choices; or if an infection probability was higher than 20% for distant contacts, the covid pandemic would have been over after a month). For parameters that strongly influence the objective function, the bounds stemming from these considerations are typically tighter than for parameters that have a small effect on the objective function. Thus, a natural approach to improve the scaling of the optimization problem is to re-map all parameters such that the bounds are [0, 1] for all parameters. This has the additional advantage that absolute and relative convergence criteria on parameter changes become the same. **Advantages:** - Straightforward - Works well in many practical applications - Scaling is independent of start values - No problems with division by zero **Disadvantages:** - Only works if all parameters have bounds - This prohibits some kinds of other constraints in optimagic **How to specify this scaling:** ```{code-block} python --- emphasize-lines: 5,6 --- res = om.minimize( fun=fun, x0=np.arange(5), algorithm="scipy_lbfgsb", bounds=om.Bounds(lower=np.zeros(5), upper=2 * np.arange(5) + 1), scaling=om.ScalingOptions(method="bounds", clipping_value=0.0), ) ``` ## Influencing the magnitude of parameters The above approaches align the scale of parameters relative to each other. However, the overall magnitude is set rather arbitrarily. For example, when dividing by start values, the magnitude of the scaled parameters is around one. When dividing by bounds, it is somewhere between zero and one. For the performance of numerical optimizers, only the relative scales are important. However, influencing the overall magnitude can be helpful to trick some optimizers into doing things they do not want to do. For example, when there is a minimal allowed initial trust region radius, increasing the magnitude of parameters allows to effectively make the trust region radius smaller. Setting the magnitude means simply adding one more entry to the scaling options. For example, if you want to scale by bounds and increase the magnitude by a factor of five: ```{code-block} python --- emphasize-lines: 6 --- res = om.minimize( fun=fun, x0=np.arange(5), algorithm="scipy_lbfgsb", bounds=om.Bounds(lower=np.zeros(5), upper=2 * np.arange(5) + 1), scaling=om.ScalingOptions(method="bounds", clipping_value=0.0, magnitude=5), ) ``` ## Remarks ### What is the `clipping_value` In all of the above heuristics, the parameter vector is divided (elementwise) by some other vector and it is possible that some entries of the divisor are zero or close to zero. The clipping value bounds the elements of the divisor away from zero. It should be set to a strictly non-zero number for the `"start_values"` and `"gradient"` approach. The `"bounds"` approach avoids division by exact zeros by construction. The `"clipping_value"` can still be used to avoid extreme upscaling of parameters with very tight bounds. However, this means that the bounds of the re-scaled problem are not exactly [0, 1] for all parameters. (scaling-default-values)= ### Default values Scaling is disabled by default. By passing `scaling=True`, we enable scaling at the default values. We use the `"start_values"` method with a `"clipping_value"` of 0.1 and a magnitude of 1.0. This is the default method because it can be used for all optimization problems and has low computational cost. We strongly recommend you read the above guidelines and choose the method that is most suitable for your problem. { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to visualize an optimization problem\n", "\n", "Plotting the criterion function of an optimization problem can answer important questions\n", "- Is the function smooth?\n", "- Is the function flat in some directions?\n", "- Should the optimization problem be scaled?\n", "- Is a candidate optimum a global one?\n", "\n", "Below we show how to make a slice plot of the criterion function." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## The simple sphere function (again)\n", "\n", "Let's look at the simple sphere function again. This time, we specify params as dictionary, but of course, any other params format (recall [pytrees](https://jax.readthedocs.io/en/latest/pytrees.html)) would work just as well. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere(params):\n", " x = np.array(list(params.values()))\n", " return x @ x" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "params = {\"alpha\": 0, \"beta\": 0, \"gamma\": 0, \"delta\": 0}\n", "bounds = om.Bounds(\n", " lower={name: -5 for name in params},\n", " upper={name: i + 2 for i, name in enumerate(params)},\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Creating a simple slice plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.slice_plot(\n", " func=sphere,\n", " params=params,\n", " bounds=bounds,\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ ":::{note}\n", "\n", "For details on using other plotting backends, see [How to change the plotting backend](how_to_change_plotting_backend.ipynb).\n", "\n", ":::" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Interpreting the plot\n", "\n", "The plot gives us the following insights:\n", " \n", "- There is no sign of local optima. \n", "- There is no sign of noise or non-differentiablities (careful, grid might not be fine enough).\n", "- The problem seems to be convex.\n", "\n", "-> We would expect almost any derivative based optimizer to work well here (which we know to be correct in that case)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using advanced options" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig = om.slice_plot(\n", " func=sphere,\n", " params=params,\n", " bounds=bounds,\n", " # selecting a subset of params\n", " selector=lambda x: [x[\"alpha\"], x[\"beta\"]],\n", " # evaluate func in parallel\n", " n_cores=4,\n", " # rename the parameters\n", " param_names={\"alpha\": \"Alpha\", \"beta\": \"Beta\"},\n", " title=\"Amazing Plot\",\n", " # number of gridpoints in each dimension\n", " n_gridpoints=50,\n", ")\n", "fig.show()" ] } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.18" } }, "nbformat": 4, "nbformat_minor": 4 } (specify-algorithm)= # How to specify and configure algorithms This how-to guide is about the mechanics of specifying and configuring optimizers in optimagic. It is not about choosing the right algorithm for your problem. For a discussion on choosing algorithms, see [this how-to guide](how_to_algorithm_selection.ipynb) There are two ways to specify and configure optimizers. The *optimagic way* and the *scipy way*. Both use the `algorithm` argument of `minimize` and `maximize` to specify an optimizer and both are super easy to use. As the name suggests, the *scipy way* is more familiar for users of scipy.optimize. The *optimagic way* adds discoverability and autocomplete. Using the *optimagic way*, you don't need to look things up in the documentation and rarely have to leave your editor, notebook or IDE. ::::{tab-set} :::{tab-item} The optimagic way :sync: optimagic ## Selecting an algorithm ```python import optimagic as om import numpy as np def fun(x): return x @ x om.minimize( fun=fun, params=np.arange(3), algorithm=om.algos.scipy_neldermead, ) ``` The algorithm is selected by passing an algorithm class. This class is usually not imported manually, but discovered using `om.algos`. After typing `om.algos.`, your editor will show you all algorithms you can choose from. ## Configuring an algorithm To configure an algorithm with advanced options, you can create an instance of the class: ```python algo = om.algos.scipy_neldermead( stopping_maxiter=100, adaptive=True, ) om.minimize( fun=fun, params=np.arange(3), algorithm=algo, ) ``` Again, you can use your editor's autocomplete to discover all options that your chosen algorithm supports. When the instance is created, the types and values of all options are checked. Should you make a mistake, you will get an error before you run your optimization. ## Advanced autocomplete in action Assume you need a gradient-free optimizer that supports bounds on the parameters. Moreover, you have a fixed computational budget, so you want to set stopping options. If you type `om.algos.`, your editor will show you all available optimizers and a list of categories you can use to filter the results. In our case, we select `GradientFree` and `Bounded`, and we could do that in any order we want. ![autocomplete_1](../_static/images/autocomplete_1.png) After selecting one of the displayed algorithms, in our case `scipy_neldermead`, the editor shows all tuning parameters of that optimizer. If you start to type `stopping`, you will see all stopping criteria that are available. ![autocomplete_2](../_static/images/autocomplete_2.png) ## Modifying an algorithm Given an algorithm, you can easily create a **modified copy** by using the `with_option` method. ```python # using copy constructors to create variants base_algo = om.algorithms.fides(stopping_maxiter=1000) algorithms = [ base_algo.with_option(trustregion_initial_radius=r) for r in [0.1, 0.2, 0.5] ] for algo in algorithms: minimize( fun=fun, params=np.arange(3), algorithm=algo, ) ``` ::: :::{tab-item} The scipy way :sync: scipy ## Selecting an algorithm ```python import optimagic as om import numpy as np def fun(x): return x @ x om.minimize( fun=fun, params=np.arange(3), algorithm="scipy_lbfgsb", ) ``` For a list of all supported algorithm names, see {ref}`list_of_algorithms`. ```{note} To provide full compatibility with scipy, you can also select algorithms with the argument `method` under their original scipy name, e.g. `method="L-BFGS-B"` instead of `algorithm="scipy_lbfgsb"`. ``` ## Configuring an algorithm To configure an algorithm, you can pass a dictionary to the `algo_options` argument. ```python options = { "stopping_maxiter": 100, "adaptive": True, } om.minimize( fun=fun, params=np.arange(3), algorithm="scipy_neldermead", algo_options=options, ) ``` If `algo_options` contains options that are not supported by the optimizer, they will be ignored and you get a warning. To find out which options are supported by an optimizer, see {ref}`list_of_algorithms`. ::: :::: (params)= # How to specify `params` `params` is the first argument of any criterion function in optimagic. It collects all the parameters to estimate, optimize, or differentiate over. In many optimization libraries, `params` must be a one-dimensional numpy array. In optimagic, it can be an arbitrary pytree (think nested dictionary) containing numbers, arrays, pandas.Series, and/or pandas.DataFrames. Below, we show a few examples of what is possible in optimagic and discuss the advantages and drawbacks of each of them. Again, we use the simple `sphere` function you know from other tutorials as an example. ```{eval-rst} .. tab-set:: .. tab-item:: Array A frequent choice of ``params`` is a one-dimensional numpy array. This is because one-dimensional numpy arrays are all that is supported by most optimizer libraries. In our opinion, it is rarely a good choice to represent parameters as flat numpy arrays and then access individual parameters or sclices by positions. The only exception are simple optimization problems with very-fast-to-evaluate criterion functions where any overhead must be avoided. If you still want to use one-dimensional numpy arrays, here is how: .. code-block:: python import optimagic as om def sphere(params): return params @ params om.minimize( fun=sphere, params=np.arange(3), algorithm="scipy_lbfgsb", ) .. tab-item:: DataFrame Originally, pandas DataFrames were the mandatory format for ``params`` in optimagic. They are still highly recommended and have a few special features. For example, they allow to bundle information on start parameters and bounds together into one data structure. Let's look at an example where we do that: .. code-block:: python def sphere(params): return (params["value"] ** 2).sum() params = pd.DataFrame( data={"value": [1, 2, 3], "lower_bound": [-np.inf, 1.5, 0]}, index=["a", "b", "c"], ) om.minimize( fun=sphere, params=params, algorithm="scipy_lbfgsb", ) DataFrames have many advantages: - It is easy to select single parameters or groups of parameters or work with the entire parameter vector. Especially, if you use a well designed MultiIndex. - It is very easy to produce publication quality LaTeX tables from them. - If you have nested models, you can easily update the parameter vector of a larger model with the values from a smaller one (e.g. to get good start parameters). - You can bundle information on bounds and values in one place. - It is easy to compare two params vectors for equality. If you are sure you won't have bounds on your parameter, you can also use a pandas.Series instead of a pandas.DataFrame. A drawback of DataFrames is that they are not JAX compatible. Another one is that they are a bit slower than numpy arrays. .. tab-item:: Dict ``params`` can also be a (nested) dictionary containing all of the above and more. .. code-block:: python def sphere(params): return params["a"] ** 2 + params["b"] ** 2 + (params["c"] ** 2).sum() res = om.minimize( fun=sphere, params={"a": 0, "b": 1, "c": pd.Series([2, 3, 4])}, algorithm="scipy_neldermead", ) Dictionarys of arrays are ideal if you want to do vectorized computations with groups of parameters. They are also a good choice if you calculate derivatives with JAX. While optimagic won't stop you, don't go too far! Having parameters in very deeply nested dictionaries makes it hard to visualize results and/or even to compare two estimation results. .. tab-item:: Scalar If you have a one-dimensional optimization problem, the natural way to represent your params is a float: .. code-block:: python def sphere(params): return params**2 om.minimize( fun=sphere, params=3, algorithm="scipy_lbfgsb", ) ``` { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# How to visualize optimizer histories\n", "\n", "optimagic's `criterion_plot` can visualize the history of function values for one or multiple optimizations. \n", "optimagic's `params_plot` can visualize the history of parameter values for one optimization. \n", "\n", "This can help you to understand whether your optimization actually converged and if not, which parameters are problematic. \n", "\n", "It can also help you to find the fastest optimizer for a given optimization problem. " ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import optimagic as om" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "## Run two optimization to get example results" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "def sphere(x):\n", " return x @ x\n", "\n", "\n", "results = {}\n", "for algo in [\"scipy_lbfgsb\", \"scipy_neldermead\"]:\n", " results[algo] = om.minimize(sphere, params=np.arange(5), algorithm=algo)" ] }, { "cell_type": "markdown", "id": "4", "metadata": {}, "source": [ "## Make a single criterion plot" ] }, { "cell_type": "code", "execution_count": null, "id": "5", "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(results[\"scipy_neldermead\"])\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "6", "metadata": {}, "source": [ ":::{note}\n", "\n", "For details on using other plotting backends, see [How to change the plotting backend](how_to_change_plotting_backend.ipynb).\n", "\n", ":::" ] }, { "cell_type": "markdown", "id": "7", "metadata": {}, "source": [ "## Compare two optimizations in a criterion plot" ] }, { "cell_type": "code", "execution_count": null, "id": "8", "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(results)\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "9", "metadata": {}, "source": [ "## Use some advanced options of criterion plot" ] }, { "cell_type": "code", "execution_count": null, "id": "10", "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(\n", " results,\n", " # cut off after 180 evaluations\n", " max_evaluations=180,\n", " # show only the current best function value\n", " monotone=True,\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "## Make a params plot" ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "fig = om.params_plot(results[\"scipy_neldermead\"])\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "13", "metadata": {}, "source": [ "## Use advanced options of params plot" ] }, { "cell_type": "code", "execution_count": null, "id": "14", "metadata": {}, "outputs": [], "source": [ "fig = om.params_plot(\n", " results[\"scipy_neldermead\"],\n", " # cut off after 180 evaluations\n", " max_evaluations=180,\n", " # select only the last three parameters\n", " selector=lambda x: x[2:],\n", ")\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "15", "metadata": {}, "source": [ "## Criterion plot with multistart optimization" ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "def alpine(x):\n", " return np.sum(np.abs(x * np.sin(x) + 0.1 * x))\n", "\n", "\n", "res = om.minimize(\n", " alpine,\n", " params=np.arange(7),\n", " bounds=om.Bounds(soft_lower=np.full(7, -3), soft_upper=np.full(7, 10)),\n", " algorithm=\"scipy_neldermead\",\n", " multistart=om.MultistartOptions(n_samples=100, convergence_max_discoveries=3),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "17", "metadata": {}, "outputs": [], "source": [ "fig = om.criterion_plot(res, max_evaluations=1000, monotone=True)\n", "fig.show()" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.17" } }, "nbformat": 4, "nbformat_minor": 5 } # Explanation This section provides background information on numerical topics and details of optimagic. It is completely optional and not necessary if you are just starting out. ```{toctree} --- maxdepth: 1 --- aggregation_level implementation_of_constraints internal_optimizers why_optimization_is_hard.ipynb explanation_of_numerical_optimizers tests_for_supported_optimizers numdiff_background ``` (aggregation_level)= # Problem types (`AggregationLevel`) optimagic can optimize three kinds of objective functions: **scalar**, **least-squares**, and **likelihood**. Internally these are represented by {class}`~optimagic.typing.AggregationLevel`. You tell optimagic which kind you have by marking the objective function with a decorator from `optimagic.mark` (`@om.mark.least_squares`, `@om.mark.likelihood`, or optionally `@om.mark.scalar`). That mark changes: 1. **What your function should return** (a single number vs a vector of contributions or residuals). 1. **Which specialized optimizers you can use** (for example pounders for least-squares, or BHHH for likelihood). 1. **How error penalties and derivatives are interpreted** when something goes wrong (see {ref}`how-to-errors`). Any marked function can still be solved with a normal scalar optimizer; optimagic aggregates the vector output when needed (sum of squares for least-squares, sum of contributions for likelihood). ## Scalar problems This is the default. Your function returns a **single number** — the value to minimize (or maximize). ```python import optimagic as om import numpy as np # @om.mark.scalar is optional; unmarked functions are treated as scalar def sphere(params): return params @ params om.minimize(sphere, params=np.arange(3), algorithm="scipy_lbfgsb") ``` Use this whenever you do not have least-squares or likelihood structure to exploit. ## Least-squares problems Mark the function with `@om.mark.least_squares` and return the **residuals** (a vector or pytree), **not** the sum of squared residuals. ```python @om.mark.least_squares def ls_sphere(params): return params # residuals; optimagic forms sum of squares if needed ``` **Why mark it?** Specialized least-squares solvers can use the residual structure and are often much faster than treating $f(x)=\sum_i r_i(x)^2$ as a black-box scalar. If you only return the scalar sum of squares, those solvers cannot be used. See {ref}`how-to-fun` for a short usage example. ## Likelihood problems Mark the function with `@om.mark.likelihood` and return a **vector (or pytree) of per-observation log-likelihood contributions**, not a single summed log-likelihood. ```python @om.mark.likelihood def loglike_contributions(params): # return one log-density value per observation (an array), not their sum ... ``` **Sign / maximize vs minimize:** return the actual log-likelihood contributions (the quantities you would sum to get $\ell(\theta)$). Prefer {func}`~optimagic.maximize` for maximum likelihood; optimagic flips the sign internally for the solver. If you prefer {func}`~optimagic.minimize`, return **negative** log-likelihood contributions instead. Do **not** return only the summed scalar log-likelihood if you want likelihood-specific optimizers — they need the contributions. For estimation workflows built on likelihood functions, see also {ref}`estimagic`. ## How this relates to `AggregationLevel` | Problem | Decorator | Typical return | `AggregationLevel` | | ------------- | ------------------------- | ------------------- | ------------------ | | Scalar | none or `@om.mark.scalar` | `float` | `SCALAR` | | Least-squares | `@om.mark.least_squares` | residual vector | `LEAST_SQUARES` | | Likelihood | `@om.mark.likelihood` | contribution vector | `LIKELIHOOD` | Optimizers are also tagged with a `solver_type` of the same enum (see {ref}`internal_optimizer_interface`). Matching the mark on your function to the solver type is what lets optimagic pick the right internal representation. (explanation-of-numerical-optimizers)= # Introduction to basic types of numerical optimization algorithms There are hundreds of different numerical optimization algorithm. However, most of them build on a few basic principles. Knowing those principles helps to classify algorithms and thus allows you to connect information about new algorithms with the stuff you already know. The main principles we describe here are: - Derivative based line search algorithms - Derivative based trust region algorithms - Derivative free trust region algorithms - Derivative free direct search algorithms This covers a large range of the algorithms that come with optimagic. In contrast, the following classes of optimizers are also accessible via optimagic, but not yet covered in this overview: - Conjugate gradient methods - Genetic algorithms - Grid or random search - Bayesian Optimization For each class of algorithms we describe the basic idea, show a gif of a stylized implementation with a graphical explanation of each iteration and a gif that shows how a real algorithm of the class converges. All of the above algorithms are local optimization algorithms that can (and will in fact) get stuck in local optima. If you need a global optimum, you will need to start them from several starting points and take the best result. ## Derivative based line search algorithms ### Basic idea 1. Use first derivative to get search direction 1. Use approximated second derivative to guess step length 1. Use a line search algorithm to see how far to go in the search direction In other words, the algorithm first fixes a promising direction and then figures out how far it should go in that direction. The important insight here is that even though the parameter space might be high dimensional, the line search problem remains one dimensional and thus simple to solve. Moreover, the line search problem is typically not solved exactly but only approximately. The exact termination conditions for the line search problem are complicated, but most of the time the initial guess for the step length is accepted. ### Stylized implementation ```{image} ../../_static/images/stylized_line_search.gif ``` ### Convergence of a real algorithm ```{image} ../../_static/images/history_l-bfgs-b.gif ``` ## Derivative based trust-region algorithms ### Basic idea 1. Fix a trust region radius 1. Construct a Taylor expansion of the function based on function value, gradient, and (approximation to) Hessian 1. Minimize the Taylor expansion within the trust region 1. Evaluate function again at the argmin of the Taylor expansion 1. Compare expected and actual improvement 1. Accept the new parameters if actual vs. expected improvement is good enough. 1. Potentially modify the trust region radius 1. Go back to 2. In other words, the algorithm first fixes a maximum step length (the trust region radius) and then figures out in which direction to go. If the surrogate model (usually a quadratic taylor expansion) approximates the function well, trust region algorithms can converge extremely fast. The main insight here is that evaluating the surrogate model is usually much cheaper than evaluating the actual criterion function and thus the trust region subproblem can be solved very fast. As can be seen in the stylized implementation, the approximation does not actually have to be very good. The only thing that matters is that it points the optimizer in the right direction. ### Stylized implementation ```{image} ../../_static/images/stylized_gradient_based_trust_region.gif ``` ### Convergence of a real algorithm ```{image} ../../_static/images/history_trust-ncg.gif ``` ## Derivative free trust region algorithms ### Basic Idea The basic idea is very similar to derivative based trust region algorithms. The only difference is that instead of a Taylor approximation which requires derivatives, we need to come up with another type of surrogate model. In order to fit this model, the algorithm evaluates the criterion function at a few points inside the trust region. Depending on how many points those are the surrogate model is a interpolation or regression model. If there are very few points it might even be an underdetermined interpolation model. In that case some kind of regularization is needed. Note that for differentiable functions without closed form derivatives, one way to define the surrogate model would be a Taylor approximation calculated from numerical derivatives. However, this would be a rather inefficient choice because points that are spaced more evenly throughout the trust region provide more information about the criterion function than the numerical derivatives. ### Stylized implementation ```{image} ../../_static/images/stylized_gradient_free_trust_region.gif ``` ### Convergence of a real algorithm ```{image} ../../_static/images/history_cobyla.gif ``` ## Derivative free direct search algorithms ### Basic Idea 1. Evaluate function at points lying in a fixed pattern around the current point 1. Accept the best point as new current point 1. Potentially modify the size or spread of the pattern 1. Go back to 1. Direct search algorithms are also called pattern search algorithms. They can typically deal well with small amounts of noise, because only the ordering of function values is used, not the magnitudes. However, they are relatively slow compared to the other algorithms. ### Stylized implementation ```{image} ../../_static/images/stylized_direct_search.gif ``` ### Convergence of a real algorithm ```{image} ../../_static/images/history_nelder-mead.gif ``` (implementation_of_constraints)= # How constraints are implemented Most of the optimizers wrapped in optimagic cannot deal natively with anything but box constraints. So the problem they can solve is: $$ \min_{x \in \mathbb{R}^k} f(x) \quad \text{s.t.} \hspace{0.5cm} l \leq x \leq u $$ However, in most econometric applications, we also need other constraints. For example, we may require that some parameters sum to a value, form a covariance matrix, or are probabilities. More abstractly, the problem becomes: $$ \min_{x \in \mathbb{R}^k} f(x) \quad \text{s.t.} \hspace{0.5cm} l \leq x \leq u \text{ and } C(x) = 0 $$ There are two basic ways of converting optimizers, which, natively, can only deal with box constraints, into constrained optimizers: Reparametrization and penalties. Below, we explain what both approaches are, why we chose the reparametrization approach over penalties, and which reparametrizations we are using for each type of constraint. In this text, we focus on constraints that can be solved by optimagic via bijective and differentiable transformations. General nonlinear constraints do not fall into this category. If you want to use nonlinear constraints, you can still do so, but optimagic will simply pass the constraints to your chosen optimizer. See {ref}`constraints` for more details. ## Possible approaches ### Reparametrizations In the reparametrization approach, we need to find an invertible mapping $g : \mathbb{R}^{k'} \to \mathbb{R}^k$, and two new bounds $l'$ and $u'$ such that: $$ l' \leq \tilde{x} \leq u' \iff l \leq g(\tilde{x}) \leq u \text { and } C(g(\tilde{x})) = 0 $$ This means that: $$ \min_{\tilde{x} \in \mathbb{R}^{k'}} f(g(\tilde{x})) \quad \text{s.t.} \hspace{0.5cm} l' \leq \tilde{x} \leq u'\\ $$ is equivalent to the original minimization problem. This sounds more complicated than it is. Let's look at the simple example of a two dimensional parameter vector, where our constraint is that the two parameters have to sum to 5. $$ x = (x_1, x_2) f(x) = x_1^2 + 2 x_2^2 c(x) = x_1 + x_2 - 5 \tilde{x} = x_1 g(\tilde{x}) = (\tilde{x}, 5 - \tilde{x}) $$ Typically, users implement such reparametrizations manually and write functions to convert between the parameters of interest and their reparametrized version. optimagic does this for you, for a large number of constraints that are typically used in econometric applications. For this approach to be efficient, it is crucial that the reparametrizations preserve desirable properties of the original problem. In particular, the mapping $g$ should be differentiable and if possible linear. Moreover, the dimensionality of $\tilde{x}$ should be chosen as small as possible. optimagic only implements constraints that can be enforced with differentiable transformations and always achieves full dimensionality reduction. ### Penalties The penalty approach is conceptually much simpler. Whenever $C(x) \neq 0$, a penalty term is added to the criterion function. If the penalty term is large enough (e.g. as large as the criterion function at the start values), this penalty ensures that any x that does not satisfy the constraints can not be optimal. While the generality and conceptual simplicity of this approach is attractive, it also has its drawbacks. Applying penalties in a naive way can introduce kinks, discontinuities, and even local optima into the penalized criterion. ## What optimagic does We chose to implement constraints via reparametrizations for the following reasons: - Reparametrizations ensure that the criterion function is only evaluated at parameters that satisfy all constraints. This is not only efficient, but essential if the criterion function is only defined for such parameters. - Reparametrizations can often achieve a substantial dimensionality reduction. In particular, fixes and equality constraints are implemented at zero cost, i.e. as efficiently as if you directly plugged them into your original problem. This is important because fixes and equality constraints often make user code much nicer and more flexible. - It is easier to preserve desirable properties such as convexity and differentiability with reparametrizations rather than penalties. The constraints that can be implemented via reparametrizations are available for all optimizers. More general constraints are only available with optimizers that can deal natively with them. This includes all optimizers from the `nlopt` and `ipopt` libraries. ## The non-trivial reparametrizations Fixed parameters, equality, and pairwise equality constraints can be implemented trivially with reparametrizations by simply plugging them into the criterion function. Increasing and decreasing constraints are internally implemented as linear constraints. The following section explains how the other types of constraints are implemented: ### Covariance and sdcorr constraints The main difficulty with covariance and sdcorr constraints is to keep the (implied) covariance matrix valid, i.e. positive semi-definite. In both cases, $\tilde{x}$ contains the non-zero elements of the lower triangular cholesky factor of the (implied) covariance matrix. For covariance constraints, $g$ is then simply the product of the cholesky factor with its transpose. For the sdcorr covariance matrix, the product is further converted to standard deviations and the unique elements of a covariance matrix. Several papers show that the cholesky reparametrization is a very efficient way to optimize over covariance matrices. Examples are {cite}`Pinheiro1996` and {cite}`Groeneveld1994`. A limitation of this approach is that there can be no additional fixes, box constraints, or other constraints on any of the involved parameters. (linear-constraint-implementation)= ### Linear constraints Assume we have m linear constraints on an n-dimensional parameter vector. Then the set of all parameter vectors that satisfies the constraints can be written as: $$ \mathbf{X} \equiv \{\mathbf{x} \in \mathbb{R}^n \mid \mathbf{l} \leq \mathbf{Ax} \leq \mathbf{u}\} $$ We are looking for a set $\mathbf{\tilde{X}}$ that only satisfies box constraints and reparametrizations. The reparametrizations will turn out to be a linear mapping, and thus have a matrix representation, say M. We are good if the following holds: $$ x \in \mathbf{X} \iff \exists \mathbf{\tilde{x}} \in \mathbf{\tilde{X}} \text{s.t.} \mathbf{x} = \mathbf{M\tilde{x}} $$ Suitable choices of $\mathbf{\tilde{X}}$ and $\mathbf{M}$ are: $$ \mathbf{\tilde{X}} \equiv \{(\tilde{x}_1, \tilde{x}_2)^T \mid \mathbf{\tilde{x}}_1 \in \mathbb{R}^{k} \text{ and } \mathbf{l} \leq \mathbf{\tilde{x}}_2 \leq \mathbf{l}\} \mathbf{M} = \left[ {\begin{array}{cc} \mathbb{I}_n[k] \\ A \\ \end{array} } \right]^{-1} $$ where $k = m - n$ and $\mathbb{I}_n[k]$ are the k rows of the identity matrix that make all rows of $\mathbf{M}$ linearly independent. **Proof:** "$\Rightarrow$": Let $x\in \mathbf{X}$, then we define $\mathbf{\tilde{x}} = \mathbf{M}^{-1} x$. Claim: $\mathbf{\tilde{x}} \in \mathbf{\tilde{X}}$: \\ $$ \mathbf{\tilde{x}} = \mathbf{M}^{-1} x = \left[ {\begin{array}{cc} \mathbb{I}_n[k]x \\ Ax \\ \end{array} } \right] = (\tilde{x}_1, \tilde{x}_2)^T $$ where $\tilde{x}_1 \in \mathbb{R}^k$ and $\mathbf{l} \leq \mathbf{\tilde{x}}_2 \leq \mathbf{u}$ because $\mathbf{l} \leq \mathbf{Ax} \leq \mathbf{u}$. Thus $\mathbf{\tilde{x}} \in \mathbf{\tilde{X}}$. "$\Leftarrow$" (Proof by negation): Let $x \not\in \mathbf{X}$ and define $\mathbf{\tilde{x}} = \mathbf{M}^{-1} x$. Claim $\mathbf{\tilde{x}} \not\in \mathbf{\tilde{X}}$. By the same argument as above we can show, that, because $\neg(\mathbf{l} \leq \mathbf{Ax} \leq \mathbf{u})$, $\mathbf{\tilde{x}} \not\in \mathbf{\tilde{X}}$. The rank condition on M makes it clear that there can be at most as many linear constraints as involved parameters. This includes any box constraints on the involved parameters. ### Probability constraints A probability constraint on k parameters means that all parameters lie in $[0, 1]$ and their sum equals one. While those are all linear constraints, they cannot be implemented in the way described above, because there are k + 1 constraints for k parameters. Instead we do the following $$ \tilde{x} = (\tilde{x}_1, \tilde{x}_2, \ldots, \tilde{x}_{k - 1})\\ g(\tilde{x}) = (\frac{\tilde{x}_1}{1 + \sum_{i=1}^{k-1}\tilde{x}_i}, \frac{\tilde{x}_2}{1 + \sum_{i=1}^{k-1}\tilde{x}_i}, \ldots, \frac{1}{1 + \sum_{i=1}^{k-1}\tilde{x}_i})\\ l' = (0, 0, \ldots, 0) $$ A limitation of this approach is that there can be no additional fixes, box constraints or other constraints on any of the involved parameters. **References** ```{eval-rst} .. bibliography:: ../refs.bib :filter: docname in docnames ``` (internal_optimizer_interface)= # Internal optimizers for optimagic optimagic provides a large collection of optimization algorithm that can be used by passing the algorithm name as `algorithm` into `maximize` or `minimize`. Advanced users can also use optimagic with their own algorithm, as long as it conforms with the internal optimizer interface. The advantages of using the algorithm with optimagic over using it directly are: - You can collect the optimizer history and create criterion_plots and params_plots. - You can use flexible formats for your start parameters (e.g. nested dicts or namedtuples) - optimagic turns unconstrained optimizers into constrained ones. - You can use logging. - You get great error handling for exceptions in the criterion function or gradient. - You get a parallelized and customizable numerical gradient if you don't have a closed form gradient. - You can compare your optimizer with all the other optimagic optimizers on our benchmark sets. All of this functionality is achieved by transforming a more complicated user provided problem into a simpler problem and then calling "internal optimizers" to solve the transformed problem. (functions_and_classes_for_internal_optimizers)= ## Functions and classes for internal optimizers The functions and classes below are everything you need to know to add an optimizer to optimagic. To see them in action look at [this guide](../how_to/how_to_add_optimizers.ipynb) ```{eval-rst} .. currentmodule:: optimagic.mark ``` ```{eval-rst} .. dropdown:: mark.minimizer The `mark.minimizer` decorator is used to provide algorithm specific information to optimagic. This information is used in the algorithm selection tool, for better error handling and for processing of the user provided optimization problem. .. autofunction:: minimizer ``` ```{eval-rst} .. currentmodule:: optimagic.optimization.internal_optimization_problem ``` ```{eval-rst} .. dropdown:: InternalOptimizationProblem The `InternalOptimizationProblem` is optimagic's internal representation of objective functions, derivatives, bounds, constraints, and more. This representation is already pretty close to what most algorithms expect (e.g. parameters and bounds are flat numpy arrays, no matter which format the user provided). .. autoclass:: InternalOptimizationProblem() :members: ``` ```{eval-rst} .. currentmodule:: optimagic.optimization.algorithm ``` ```{eval-rst} .. dropdown:: InternalOptimizeResult This is what you need to create from the output of a wrapped algorithm. .. autoclass:: InternalOptimizeResult :members: ``` ```{eval-rst} .. dropdown:: Algorithm .. autoclass:: Algorithm :members: :exclude-members: with_option_if_applicable ``` (naming-conventions)= ## Naming conventions for algorithm specific arguments To make switching between different algorithm as simple as possible, we align the names of commonly used convergence and stopping criteria. We also align the default values for stopping and convergence criteria as much as possible. ```{eval-rst} You can find the harmonized names and value here: :ref:`algo_options`. ``` To align the names of other tuning parameters as much as possible with what is already there, simple have a look at the optimizers we already wrapped. For example, if you are wrapping a bfgs or lbfgs algorithm from some libray, try to look at all existing wrappers of bfgs algorithms and use the same names for the same options. ## Algorithms that parallelize Algorithms that evaluate the objective function or derivatives in parallel should only do so via `InternalOptimizationProblem.batch_fun`, `InternalOptimizationProblem.batch_jac` or `InternalOptimizationProblem.batch_fun_and_jac`. If you parallelize in any other way, the automatic history collection will stop to work. In that case, call `om.mark.minimizer` with `disable_history=True`. In that case you can either do your own history collection and add that history to `InternalOptimizeResult` or the user has to rely on logging. ## Nonlinear constraints (to be written) # Numerical differentiation: methods In this section we explain the mathematical background of forward, backward and central differences. The main ideas in this chapter are taken from {cite}`Dennis1996`. x is used for the pandas DataFrame with parameters. We index the entries of x as a n-dimensional vector, where n is the number of variables in params_sr. The forward difference for the gradient is given by: $$ \nabla f(x) = \begin{pmatrix}\frac{f(x + e_0 * h_0) - f(x)}{h_0}\\ \frac{f(x + e_1 * h_1) - f(x)}{h_1}\\.\\.\\.\\ \frac{f(x + e_n * h_n) - f(x)}{h_n} \end{pmatrix} $$ The backward difference for the gradient is given by: $$ \nabla f(x) = \begin{pmatrix}\frac{f(x) - f(x - e_0 * h_0)}{h_0}\\ \frac{f(x) - f(x - e_1 * h_1)}{h_1}\\.\\.\\.\\ \frac{f(x) - f(x - e_n * h_n)}{h_n} \end{pmatrix} $$ The central difference for the gradient is given by: $$ \nabla f(x) = \begin{pmatrix}\frac{f(x + e_0 * h_0) - f(x - e_0 * h_0)}{2 h_0}\\ \frac{f(x + e_1 * h_1) - f(x - e_1 * h_1)}{2 h_1}\\.\\.\\.\\ \frac{f(x + e_n * h_n) - f(x - e_n * h_n)}{2 h_n} \end{pmatrix} $$ For the optimal stepsize h the following rule of thumb is applied: $$ h_i = (1 + |x[i]|) * \sqrt\epsilon $$ With the above in mind it is easy to calculate the Jacobian matrix. The calculation of the finite difference w.r.t. each variable of params_sr yields a vector, which is the corresponding column of the Jacobian matrix. The optimal stepsize remains the same. For the Hessian matrix, we repeatedly call the finite differences functions. As we allow for central finite differences in the second order derivative only, the deductions for forward and backward, are left to the interested reader: $$ f_{i,j}(x) = &\frac{f_i(x + e_j * h_j) - f_i(x - e_j * h_j)}{h_j} \\ = &\frac{\frac{f(x + e_j * h_j + e_i * h_i) - f(x + e_j * h_j - e_i * h_i)}{h_i} - \frac{ f(x - e_j * h_j + e_i * h_i) - f(x - e_j * h_j - e_i * h_i) }{h_i}}{h_j} \\ = &\frac{ f(x + e_j * h_j + e_i * h_i) - f(x + e_j * h_j - e_i * h_i) }{h_j * h_i} \\ &+ \frac{ - f(x - e_j * h_j + e_i * h_i) + f(x - e_j * h_j - e_i * h_i) }{h_j * h_i} $$ For the optimal stepsize a different rule is used: $$ h_i = (1 + |x[i]|) * \sqrt[3]\epsilon $$ Similar deviations lead to the elements of the Hessian matrix calculated by backward and central differences. **References:** ```{eval-rst} .. bibliography:: ../refs.bib :filter: docname in docnames ``` # How supported optimization algorithms are tested optimagic provides a unified interface that supports a large number of optimization algorithms from different libraries. Additionally, it allows putting constraints on the optimization problem. To test the external interface of all supported algorithms, we consider different criterion (benchmark) functions and test each algorithm with every type of constraint. ## Benchmark functions for testing ### Trid function > $f({x}) = \Sigma^{D}_{i=1}(x_{i} - 1)^2 - \Sigma^{D}_{i=2}(x_i x_{i-1})$ ### Rotated Hyper Ellipsoid function > $f({x}) = \Sigma^{D}_{i=1} \Sigma^{i}_{j=1}x_j^2$ ### Rosenbrock function > $\Sigma^{D-1}_{i=1}(100(x_i+1 - x_i^2)^2 + (x_i - 1)^2)$ ### Sphere function > $f({x}) = \Sigma^{D}_{i=1} ix_{i}^2$ ## How testcases are implemented We consider different implementations of each criterion and its gradient. All algorithms accept criterion functions specified in a dictionary, while a subset also accepts the criterion specified in scalar form. Likewise, if specified, the gradient of a criterion can be an np.ndarray or a pandas object. We test for all possible cases. For instance, for rotated hyper ellipsoid, we implement the following functions: - rotated_hyper_ellipsoid_scalar_criterion - rotated_hyper_ellipsoid_dict_criterion: This provides a dictionary wherein the `contributions` and `root_contributions` keys present the criterion as a least squares problem, relevant when we are testing a least squares algorithm. - rotated_hyper_ellipsoid_gradient - rotated_hyper_ellipsoid_pandas_gradient: Computes the gradient of the rotated hyper ellipsoid function, as a pandas object. - rotated_hyper_ellipsoid_criterion_and_gradient These criterion functions are specified in the `examples` directory. For an overview of all constraints supported in optimagic, please see [this how-to guide]. We write several test functions, each corresponding to the case of one constraint. Given the constraint, the test function considers all possible combinations of the algorithm, whether to maximize or to minimize, criterion function implementation, gradient implementation for that criterion (if provided), and whether `criterion_and_derivative` has been provided or not. Below, we show the calculations behind the true values, for each testcase (one criterion and one constraint). ### Trid: Solutions for three-dimension case > $f({x}) = (x_1-1)^2 + (x_2-1)^2 + (x_3-1)^2 - x_2 x_1 - x_3 x_2$ ```{eval-rst} .. dropdown:: No constraints .. code-block:: python constraints = [] :math:`x* = (3, 4, 3)` ``` ```{eval-rst} .. dropdown:: Fixed constraints .. code-block:: python constraints = [{"loc": "x_1", "type": "fixed", "value": 1}] :math:`x_{1} = 1 \rightarrow f(x) = (x_2 - 1)^2 + (x_3 - 1)^2 - x_2 - x_3 x_2 \\ \Rightarrow \frac{\delta f({x})}{\delta x_2} = 2x_2 - 3 - x_3 = 0 \Rightarrow x_3 = 2x_2 - 3\\ \Rightarrow \frac{\delta f({x})}{\delta x_3} = 2x_3 - 2 - x_2 = 0 \Rightarrow x_2 = 2x_3 - 2\\ \Rightarrow x_2 = \frac{8}{3} , \quad x_3 = \frac{7}{3}\\ \rightarrow x* = (1,\frac{8}{3}, \frac{7}{3})` ``` ```{eval-rst} .. dropdown:: Probability constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2"], "type": "probability"}] :math:`x_{1} + x_{2} = 1, \quad 0 \leq x_1 \leq 1, \quad 0 \leq x_2 \leq 1 \\ \rightarrow f({x}) = 3x_1^2 - 3x_1 - 3x_3 + x_3^2 + x_1 x_3 + 2 \\ \Rightarrow \frac{\delta f({x})}{\delta x_1} = 6x_1 - 3 + x_3 = 0 \Rightarrow x_3 = 3 - 6x_1\\ \Rightarrow \frac{\delta f({x})}{\delta x_3} = 2x_3 - 3 + x_1 = 0 \Rightarrow x_1 = 3 - 2x_3\\ \Rightarrow x_1 = \frac{3}{11}, \quad x_3 = \frac{15}{11}\\ \rightarrow x* = (\frac{3}{11}, \frac{8}{11}, \frac{15}{11})` ``` ```{eval-rst} .. dropdown:: Increasing constraint .. code-block:: python constraints = [{"loc": ["x_2", "x_3"], "type": "increasing"}] :math:`\mathcal{L}({x_i}) = (x_1 - 1)^2 + (x_2 - 1)^2 + (x_3 - 1)^2 - x_1 x_2 - x_3 x_2 - \lambda(x_3 - x_2)\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_1} = 2(x_1 - 1) - x_2 = 0\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_2} = 2(x_2 - 1) - x_1 - x_3 + \lambda = 0\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_3} = 2(x_3 - 1) - x_2 - \lambda = 0\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta \lambda} = - x_3 + x_2 = 0\\ \Rightarrow x_2 = 2(x_1 - 1) = x_3 = \frac{10}{3}\\ \Rightarrow 2(x_2 - 1) - x_1 - 2 = 0\\ \Rightarrow 4(x_1 - 1) - 2 - x_1 - 2 = 0\\ \Rightarrow 3x_1 - 8 = 0 \Rightarrow x_1 = \frac{8}{3}\\ \rightarrow x* = (\frac{8}{3}, \frac{10}{3}, \frac{10}{3})` ``` ```{eval-rst} .. dropdown:: Decreasing constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2"], "type": "decreasing"}] Solution unavailable. ``` ```{eval-rst} .. dropdown:: Equality constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "equality"}] :math:`x_{1} = x_{2} = x_{3} = x \\ \rightarrow f({x}) = x^2 - 6x + 3\\ \Rightarrow \frac{\delta f({x})}{\delta x} = 2x - 6 = 0\\ \Rightarrow x = 3\\ \rightarrow x* = (3,3,3)` ``` ```{eval-rst} .. dropdown:: Pairwise equality constraint .. code-block:: python constraints = [{"locs": ["x_1", "x_2"], "type": "pairwise_equality"}] :math:`x_{1} = x_{2} \\ \rightarrow f({x}) = 2(x_1 - 1)^2 + (x_3 - 1)^2 - x_1^2 - x_3 x_1\\ \Rightarrow \frac{\delta f({x})}{\delta x_1} = 2x_1 - x_3 - 4 = 0 \Rightarrow x_3 = 2x_1 - 4\\ \Rightarrow \frac{\delta f({x})}{\delta x_3} = 2x_3 - x_1 - 2 = 0 \Rightarrow x_1 = 2x_3 - 2\\ \Rightarrow x_1 = \frac{10}{3}, x_3 = \frac{8}{3}\\ \rightarrow x* = (\frac{10}{3},\frac{10}{3},\frac{8}{3})` ``` ```{eval-rst} .. dropdown:: Covariance constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "covariance"}] Solution unavailable. ``` ```{eval-rst} .. dropdown:: sdcorr constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "sdcorr"}] Solution unavailable. ``` ```{eval-rst} .. dropdown:: Linear constraint .. code-block:: python constraints = [{"loc": ["x_1", "x_2"], "type": "linear", "weights": [1, 2], "value": 4}] :math:`x_1 + 2x_2 = 4\\ \mathcal{L}({x_i}) = (x_1 - 1)^2 + (x_2 - 1)^2 + (x_3 - 1)^2 - x_1 x_2 - x_3 x_2 - \lambda(x_1 +2x_2-4)\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_1} = 2(x_1 - 1) - x_2 - \lambda = 0\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_2} = 2(x_2 - 1) - x_1 - x_3 - 2\lambda = 0\\ \Rightarrow \frac{\delta \mathcal{L}}{\delta x_3} = 2(x_3 - 1) - x_2 = 0 \\ \Rightarrow \frac{\delta \mathcal{L}}{\delta \lambda} = - x_1 - 2x_2 + 4 = 0\\ \Rightarrow x_2 = 2(x_3 - 1), \quad x_1 = 4 - 2x_2\\ \Rightarrow 2(4 - 2x_2 - 1) - x_2 = x_2 - 1 - 2 + x_2 - \frac{x_2}{4} - \frac{1}{2}\\ \rightarrow x* = (\frac{32}{27}, \frac{38}{27}, \frac{46}{27})` ``` ### Rotated Hyper Ellipsoid: Solutions for three-dimension case > $f({x}) = x^2_1 + (x^2_1 + x^2_2) + (x^2_1 + x^2_2 + x^2_3)$ > > > ```{eval-rst} > > .. dropdown:: No constraints > > > > .. code-block:: python > > > > constraints = [] > > > > :math:`x* = (0, 0, 0)` > > ``` > > > > ```{eval-rst} > > .. dropdown:: Fixed constraints > > > > .. code-block:: python > > > > constraints = [{"loc": "x_1", "type": "fixed", "value": 1}] > > > > :math:`x_{1} = 1 > > \rightarrow x* = (1, 0, 0)` > > ``` > > > > ```{eval-rst} > > .. dropdown:: Probability constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2"], "type": "probability"}] > > > > :math:`x_{1} + x_{2} = 1, \quad 0 \leq x_1 \leq 1, \quad 0 \leq x_2 \leq 1 \\ > > \mathcal{L}({x_i}) = x^2_1 + (x^2_1 + x^2_2) + (x^2_1 + x^2_2 + x^2_3)\\ > > -\lambda(x_1 +x_2-1)\\ > > \Rightarrow \frac{\delta \mathcal{L}}{\delta x_1}\\ > > = 6x_1 - \lambda = 0\\ > > \Rightarrow \frac{\delta \mathcal{L}}{\delta x_2}\\ > > = 4x_2 - \lambda = 0\\ > > \Rightarrow \frac{\delta \mathcal{L}}{\delta x_3}\\ > > = 2 x_3 = 0\\ > > \Rightarrow \frac{\delta \mathcal{L}}{\delta \lambda} \\ > > = -x_1 - x_2 + 1 = 0\\ > > \rightarrow x* = (\frac{2}{5}, \frac{3}{5}, 0),\\ > > \quad f({x*}) = \frac{6}{5}` > > ``` > > > > ```{eval-rst} > > .. dropdown:: Increasing constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_2", "x_3"], "type": "increasing"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: Decreasing constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2"], "type": "decreasing"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: Equality constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "equality"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: Pairwise equality constraints > > > > .. code-block:: python > > > > constraints = [{"locs": ["x_1", "x_2"], "type": "pairwise_equality"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: Covariance constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "covariance"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: sdcorr constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "sdcorr"}] > > > > Not binding :math:`\rightarrow x* = (0, 0, 0)` > > > > ``` > > > > ```{eval-rst} > > .. dropdown:: Linear constraints > > > > .. code-block:: python > > > > constraints = [{"loc": ["x_1", "x_2"], "type": "linear", "weights": [1, 2], "value": 4}] > > > > :math:`x_1 + 2x_2 = 4\\\mathcal{L}({x_i}) = x^2_1 + (x^2_1 + x^2_2) + > > (x^2_1 + x^2_2 + x^2_3) -\lambda(x_1 +2x_2-4)\\ > > \Rightarrow \frac{\delta\mathcal{L}}{\delta x_1} = 6x_1 - \lambda = 0\\ > > \Rightarrow \frac{\delta \\ > > \mathcal{L}}{\delta x_2} = 4x_2 - 2\lambda = 0\\ > > \Rightarrow \frac{\delta \\ > > \mathcal{L}}{\delta x_3} = 2 x_3 = 0\\ > > \Rightarrow \frac{\delta \\ > > \mathcal{L}}{\delta \lambda} = -x_1 - 2x_2 + 4 = 0\\ > > \rightarrow x* = (\frac{4}{7}, \frac{12}{7}, 0)` > > > > > > > > > > > > > > ``` ### Rosenbrock: Solutions for three-dimension case > $f({x}) = 100(x_2 - x_1^2) + (x_1 - 1)^2$ Global minima: $x* = (1, 1, 1)$ > ```{eval-rst} > .. dropdown:: No constraints > > .. code-block:: python > > constraints = [] > > :math:`x* = (1, 1, 1)` > > ``` > > ```{eval-rst} > .. dropdown:: Fixed constraints > > .. code-block:: python > > constraints = [{"loc": "x_1", "type": "fixed", "value": 1}] > > :math:`x_{1} = 1 \rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: Fixed constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2"], "type": "probability"}] > > No solution available. > ``` > > ```{eval-rst} > .. dropdown:: Increasing constraints > > .. code-block:: python > > constraints = [{"loc": ["x_2", "x_3"], "type": "increasing"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > > ``` > > ```{eval-rst} > .. dropdown:: Decreasing constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2"], "type": "decreasing"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: Equality constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "equality"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: Pairwise equality constraints > > .. code-block:: python > > constraints = [{"locs": ["x_1", "x_2"], "type": "pairwise_equality"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: Covariance constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "covariance"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: sdcorr constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2", "x_3"], "type": "sdcorr"}] > > Not binding :math:`\rightarrow x* = (1, 1, 1)` > ``` > > ```{eval-rst} > .. dropdown:: Linear constraints > > .. code-block:: python > > constraints = [{"loc": ["x_1", "x_2"], "type": "linear", "weights": [1, 2], "value": 4}] > > No solution available. > ``` [this how-to guide]: ../how_to/how_to_constraints.md { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Why optimization is difficult\n", "\n", "This tutorial shows why optimization is difficult and why you need some knowledge in order to solve optimization problems efficiently. It is meant for people who have no previous experience with numerical optimization and wonder why there are so many optimization algorithms and still none that works for all problems. For each potential problem we highlight, we also give some ideas on how to solve it. \n", "\n", "\n", "If you simply want to learn the mechanics of doing optimization with optimagic, check out the [quickstart guide](../tutorials/optimization_overview.ipynb)\n", "\n", "\n", "The take-home message of this notebook can be summarized as follows:\n", "\n", "- The only algorithms that are guaranteed to solve all problems are grid search or other algorithms that evaluate the criterion function almost everywhere in the parameter space.\n", "- If you have more than a hand full of parameters, these methods would take too long.\n", "- Thus, you have to know the properties of your optimization problem and have knowledge about different optimization algorithms in order to choose the right algorithm for your problem. \n", "\n", "This tutorial uses variants of the sphere function from the [quickstart guide](../tutorials/optimization_overview.ipynb)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import seaborn as sns\n", "\n", "import optimagic as om" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere(x):\n", " return x @ x\n", "\n", "\n", "def sphere_gradient(x):\n", " return 2 * x" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Why grid search is infeasible\n", "\n", "Sampling based optimizers and grid search require the parameter space to be bounded in all directions. Let's assume we know that the optimum of the sphere function lies between -0.5 and 0.5, but don't know where it is exactly. \n", "\n", "In order to get a precision of 2 digits with grid search, we require the following number of function evaluations (depending on the number of parameters):" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dimensions = np.arange(10) + 1\n", "n_evals = 100**dimensions\n", "sns.lineplot(x=dimensions, y=n_evals);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you have 10 dimensions and evaluating your criterion function takes one second, you need about 3 billion years on a 1000 core cluster. Many of the real world criterion functions have hundreds of parameters and take minutes to evaluate once. This is called the curse of dimensionality." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Sampling based algorithms typically fix the number of criterion evaluations and apply them a bit smarter than algorithms that rummage the search space randomly. However, these smart tricks only work under additional assumptions. Thus, either you need to make assumptions on your problem or you will get the curse of dimensionality through the backdoor again. For easier analysis, assume we fix the number of function evaluations in a grid search instead of a sampling based algorithm and want to know which precision we can get, depending on the dimension:\n", "\n", "For 1 million function evaluations, we can expect the following precision:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dimensions = np.arange(10) + 1\n", "precision = 1e-6 ** (1 / dimensions)\n", "sns.lineplot(x=dimensions, y=precision);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How derivatives can solve the curse of dimensionality\n", "\n", "Derivative based methods do not try to evaluate the criterion function everywhere in the search space. Instead, they start at some point and go \"downhill\" from there. The gradient of the criterion function indicates which direction is downhill. Then there are different ways of determining how far to go in that direction. The time it takes to evaluate a derivative increases at most linearly in the number of parameters. Using the derivative information, optimizers can often find an optimum with very few function evaluations." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## How derivative based methods can fail\n", "\n", "To see how derivative based methods can fail, we use simple modifications of the sphere function. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(seed=0)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def sphere_with_noise(x, rng):\n", " return sphere(x) + rng.normal(scale=0.02)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "start_params = np.arange(5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grid = np.linspace(-1, 1, 1000)\n", "sns.lineplot(\n", " x=grid,\n", " y=(grid**2) + rng.normal(scale=0.02, size=len(grid)),\n", ");" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=sphere_with_noise,\n", " params=start_params,\n", " algorithm=\"scipy_lbfgsb\",\n", " logging=False,\n", " fun_kwargs={\"rng\": rng},\n", ")\n", "\n", "res.success" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.message" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So the algorithm failed, but at least tells you that it did not succed. Let's look at a different kind of numerical noise that could come from rounding. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def piecewise_constant_sphere(x):\n", " return sphere(x.round(2))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sns.lineplot(x=grid, y=grid.round(2) ** 2);" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res = om.minimize(\n", " fun=piecewise_constant_sphere,\n", " params=start_params,\n", " algorithm=\"scipy_lbfgsb\",\n", ")\n", "\n", "res" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This time, the algorithm failed silently." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 4 } # optimagic API ```{eval-rst} .. currentmodule:: optimagic ``` (maximize-and-minimize)= ## Optimization ```{eval-rst} .. dropdown:: maximize .. autofunction:: maximize ``` ```{eval-rst} .. dropdown:: minimize .. autofunction:: minimize ``` ```{eval-rst} .. dropdown:: slice_plot .. autofunction:: slice_plot ``` ```{eval-rst} .. dropdown:: criterion_plot .. autofunction:: criterion_plot ``` ```{eval-rst} .. dropdown:: params_plot .. autofunction:: params_plot ``` ```{eval-rst} .. dropdown:: OptimizeResult .. autoclass:: OptimizeResult :members: ``` ```{eval-rst} .. dropdown:: Bounds .. autoclass:: Bounds :members: ``` ```{eval-rst} .. dropdown:: Constraints .. autoclass:: FixedConstraint :members: .. autoclass:: IncreasingConstraint :members: .. autoclass:: DecreasingConstraint :members: .. autoclass:: EqualityConstraint :members: .. autoclass:: ProbabilityConstraint :members: .. autoclass:: PairwiseEqualityConstraint :members: .. autoclass:: FlatCovConstraint :members: .. autoclass:: FlatSDCorrConstraint :members: .. autoclass:: LinearConstraint :members: .. autoclass:: NonlinearConstraint :members: ``` ```{eval-rst} .. dropdown:: NumdiffOptions .. autoclass:: NumdiffOptions :members: ``` ```{eval-rst} .. dropdown:: MultistartOptions .. autoclass:: MultistartOptions :members: ``` ```{eval-rst} .. dropdown:: ScalingOptions .. autoclass:: ScalingOptions :members: ``` ```{eval-rst} .. dropdown:: LogOptions .. autoclass:: SQLiteLogOptions :members: ``` ```{eval-rst} .. dropdown:: History .. autoclass:: History :members: ``` ```{eval-rst} .. dropdown:: count_free_params .. autofunction:: count_free_params ``` ```{eval-rst} .. dropdown:: check_constraints .. autofunction:: check_constraints ``` (first_derivative)= ## Derivatives ```{eval-rst} .. dropdown:: first_derivative .. autofunction:: first_derivative ``` ```{eval-rst} .. dropdown:: second_derivative .. autofunction:: second_derivative ``` (benchmarking)= ## Benchmarks ```{eval-rst} .. dropdown:: get_benchmark_problems .. autofunction:: get_benchmark_problems ``` ```{eval-rst} .. dropdown:: run_benchmark .. autofunction:: run_benchmark ``` ```{eval-rst} .. dropdown:: profile_plot .. autofunction:: profile_plot ``` ```{eval-rst} .. dropdown:: convergence_plot .. autofunction:: convergence_plot ``` (logreading)= ## Log reading ```{eval-rst} .. dropdown:: OptimizeLogReader .. autoclass:: OptimizeLogReader ``` ## Other: ```{toctree} --- maxdepth: 1 --- utilities algo_options batch_evaluators typing ``` (utilities)= # Utility functions ```{eval-rst} .. automodule:: optimagic.utilities :members: ``` (algo_options)= # The default algorithm options ```{eval-rst} .. automodule:: optimagic.optimization.algo_options :members: ``` (batch_evaluators)= # Batch evaluators ```{eval-rst} .. automodule:: optimagic.batch_evaluators :members: ``` (typing)= # Types ```{eval-rst} .. automodule:: optimagic.typing :members: ``` # Development ```{toctree} --- maxdepth: 1 --- code_of_conduct how_to_contribute styleguide enhancement_proposals credits changes ``` (coc)= ## Code of Conduct The optimagic project has a [Code of Conduct][conduct] to which all contributors must adhere. See details in the [written policy statement][conduct]. [conduct]: https://github.com/optimagic-dev/optimagic/blob/main/.github/CODE_OF_CONDUCT.md (changes)= ```{include} ../../../CHANGES.md ``` # Credits ## The optimagic Team ```{eval-rst} +---------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ + .. figure:: ../_static/images/janos.jpg + .. figure:: ../_static/images/mariam.jpg + .. figure:: ../_static/images/tim.jpeg + .. figure:: ../_static/images/klara.jpg + + :width: 120px + :width: 120px + :width: 120px + :width: 120px + + + + + + + `Janoś Gabler `_ + `Mariam Petrosyan `_ + `Tim Mensinger `_ + `Klara Röhrl `_ + +---------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ + .. figure:: ../_static/images/tobi.png + .. figure:: ../_static/images/annica.jpeg + .. figure:: ../_static/images/sebi.jpg + .. figure:: ../_static/images/bahar.jpg + + :width: 120px + :width: 120px + :width: 120px + :width: 120px + + + + + + + `Tobias Raabe `_ + `Annica Gehlen `_ + `Sebastian Gsell `_ + `Bahar Coskun `_ + +---------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ + .. figure:: ../_static/images/aida.jpg + .. figure:: ../_static/images/hmg.jpg + .. figure:: ../_static/images/ken.jpeg + + + :width: 120px + :width: 120px + :width: 120px + + + + + + + + `Aida Takhmazova `_ + `Hans-Martin von Gaudecker `_ + `Kenneth L. Judd `_ + + +---------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+-------------------------------------------------------------------+ ``` Janoś is the original developer and architect behind optimagic (formerly estimagic). All team members are active contributors in terms of commits, advice or community building. Hans-Martin and Ken support optimagic with funding and their expertise. ## Contributors We are grateful for many contributions from the community. In particular, we want to thank Moritz Mendel, Max Blesch, Christian Zimpelmann, Robin Musolff, Sofia Badini, Sofya Akimova, Xuefei Han, Leiqiong Wan, Andrew Souther, Luis Calderon, Linda Maokomatanda, Madhurima Chandra, and Vijaybabu Gangaprasad. ## Acknowledgements We thank all institutions that have funded or supported optimagic (formerly estimagic) ```{image} ../_static/images/aai-institute-logo.svg --- width: 185px --- ``` ```{image} ../_static/images/numfocus_logo.png --- width: 200 --- ``` ```{image} ../_static/images/tra_logo.png --- width: 240px --- ``` ```{image} ../_static/images/hoover_logo.png --- width: 192px --- ``` ```{image} ../_static/images/transferlab-logo.svg --- width: 420px --- ``` # Enhancement Proposals optimagic Enhancement Proposals (EPs) can be used to discuss and design large changes. EP-00 details the EP process, the optimagic governance model and the optimagic Code of Conduct. It is the only EP that gets continuously updated. These EPs are currently in place: ```{toctree} --- maxdepth: 1 --- ep-00-governance-model.md ep-01-pytrees.md ep-02-typing.md ep-03-alignment.md ``` (ep-00)= # EP-00: Governance model & code of conduct ```{eval-rst} +------------+------------------------------------------------------------------+ | Author | `Maximilian Blesch `_, | | | `Janoś Gabler `_, | | | `Hans-Martin von Gaudecker `_, | | | `Annica Gehlen `_, | | | `Sebastian Gsell `_, | | | `Tim Mensinger `_, | | | `Mariam Petrosyan `_, | | | `Tobias Raabe `_, | | | `Klara Röhrl `_ | +------------+------------------------------------------------------------------+ | Status | Accepted | +------------+------------------------------------------------------------------+ | Type | Standards Track | +------------+------------------------------------------------------------------+ | Created | 2022-04-28 | +------------+------------------------------------------------------------------+ | Resolution | | +------------+------------------------------------------------------------------+ ``` ## Purpose This document formalizes the optimagic code of conduct and governance model. In case of changes, this document can be updated following the optimagic Enhancement Proposal process detailed below. ```{include} ../../../CODE_OF_CONDUCT.md ``` ## optimagic governance model ### Summary The governance model strives to be lightweight and based on [consensus](https://numpy.org/doc/stable/dev/governance/governance.html#consensus-based-decision-making-by-the-community) of all interested parties. Most work happens in GitHub issues and pull requests (regular decision process). Any interested party can voice their concerns or veto on proposed changes. If this happens, the optimagic Enhancement Proposal (EP) process can be used to iterate over proposals until consesus is reached (controversial decision process). If necessary, members of the steering council can moderate heated debates and help to broker a consensus. ### Regular decision process Most changes to optimagic are additions of new functionality or strict improvements of existing functionality. Such changes can be discussed in GitHub issues and discussions and implemented in pull requests. They do not require an optimagic Enhancement Proposal. Before starting to work on optimagic, contributors should read [how to contribute](how-to) and the [styleguide](styleguide). They can also reach out to existing contributors if any help is needed or anything remains unclear. We are all happy to help onboarding new contributors in any way necessary. For example, we have given introductions to git and GitHub in the past to help people make a contribution to optimagic. Pull requests should be opened as soon as work is started. They should contain a good description of the planned work such that any interested party can participate in the discussion around the changes. If planned changes turn out to be controversial, their design should be discussed in an optimagic Enhancement Proposal before the actual work starts. When the work is finished, the author of a pull request can request a review. In most cases, previous discussions will show who is a suitable reviewer. If in doubt, tag [janosg](https://github.com/janosg). Pull requests can be merged if there is at least one approving review. Reviewers should be polite, welcoming and helpful to the author of the pull request who might have spent many hours working on the changes. Authors of pull requests should keep in mind that reviewers' time is valuable. Major points should be discussed publicly on GitHub, but very critical feedback or small details can be moved to private discussions — if the latter are necessary at all (see [the bottom section of this blog post](https://rgommers.github.io/2019/06/the-cost-of-an-open-source-contribution/) for an excellent discussion of the burden that review comments place on maintainers, which might not always be obvious). Video calls can help if a discussion gets stuck. The code of conduct applies to all interactions related to code reviews. ### optimagic Enhancement Proposals (EPs) / Controversial decision process Large changes to optimagic can be proposed in optimagic Enhancement Proposals, short EPs. They serve the purpose of summarising discussions that may happen in chats, issues, pull requests, in person, or by any other means. Simple extensions (like adding new optimizers) do not need to be discussed with such a formal process. EPs are written as markdown documents that become part of the documentation. Opening an EP means opening a pull request that adds the markdown document to the documentation. It is not necessary to already have a working implementations for the planned changes, even though it might be a good idea to have rough prototypes for solutions to the most challenging parts. If the author of an EP feels that it is ready to be accepted they need to make a post in the relevant [Zulip topic](https://ose.zulipchat.com) and a comment on the PR that contains the following information: 1. Summary of all contentious aspects of the EP and how they have been resolved 1. Every interested party has seven days to comment on the PR proposing the EP, either with approval or objections. While only objections are relevant for the decision making process, approvals are a good way to signal interest in the planned change and recognize the work of the authors. 1. If there are no unresolved objections after seven days, the EP will automatically be accepted and can be merged. Note that the pull requests that actually implement the proposed enhancements still require a standard review cycle. ### Steering Council The optimagic Steering Council consists of five people who take responsibility for the future development of optimagic and the optimagic community. Being a member of the steering council comes with no special rights. The main roles of the steering council are: - Facilitate the growth of optimagic and the optimagic community by organizing community events, identifying funding opportunities and improving the experience of all community members. - Develop a roadmap, break down large changes into smaller projects and find contributors to work on the implementation of these projects. - Ensure that new contributors are onboarded and assisted and that pull requests are reviewed in a timely fashion. - Step in as moderators when discussions get heated, help to achieve consensus on controversial topics and enforce the code of conduct. The Steering Council is elected by the optimagic community during a community meeting. Candidates need to be active community members and can be nominated by other community members or themselves until the start of the election. Nominated candidates need to accept the nomination before the start of the election. If there are only five candidates, the Steering Council is elected by acclamation. Else, every participant casts five votes. The 5 candidates with the most votes become elected. Candidates can vote for themselves. Ties are resolved by a second round of voting where each participant casts as many votes as there are positions left. Remaining ties are resolved by randomization. Current memebers of the optimagic Steering Council are: - [Janoś Gabler](https://github.com/janosg) - [Annica Gehlen](https://github.com/amageh) - [Hans-Martin von Gaudecker](https://github.com/hmgaudecker) - [Tim Mensinger](https://github.com/timmens) - [Mariam Petrosyan](https://github.com/mpetrosian) ### Community meeting Community meetings can be held to elect a steering council, make changes to the governance model or code of conduct, or to make other decisions that affect the community as a whole. Moreover, they serve to keep the community updated about the development of optimagic and get feedback. Community meetings need to be announced via our public channels (e.g. the [zulip workspace](https://ose.zulipchat.com) or GitHub discussions) with sufficient time until the meeting. The definition of sufficient time will increase with the size of the community. (eppytrees)= # EP-01: Pytrees ```{eval-rst} +------------+------------------------------------------------------------------+ | Author | `Janos Gabler `_ | +------------+------------------------------------------------------------------+ | Status | Accepted | +------------+------------------------------------------------------------------+ | Type | Standards Track | +------------+------------------------------------------------------------------+ | Created | 2022-01-28 | +------------+------------------------------------------------------------------+ | Resolution | | +------------+------------------------------------------------------------------+ ``` ## Abstract This EEP explains how we will use pytrees to allow for more flexible specification of parameters for optimization or differentiation, more convenient ways of writing moment functions for msm estimation and more. The actual code to work with pytrees is implemented in [Pybaum], developed by {ghuser}`janosg` and {ghuser}`tobiasraabe`. ## Backwards compatibility All changes are fully backwards compatible. ## Motivation Estimagic has many functions that require user written functions as inputs. Examples are: - criterion functions and their derivatives for optimization - functions of which numerical derivatives are taken - functions that calculate simulated moments - functions that calculate bootstrap statistics In all cases, there are some restrictions on possible inputs and outputs of the user written functions. For example, parameters for numerical optimization need to be provided as pandas.DataFrame with a `"value"` column. Simulated moments and bootstrap statistics need to be returned as a pandas.Series, etc. Pytrees allow to relax many of those restrictions on interfaces of user provided functions. This is not only more convenient for users, but sometimes also allows to reduce overhead because the user can choose optimal data structures for their problem. ## Background: What is a pytree Pytree is a term used in TensorFlow and JAX to refer to a tree-like structure built out of container-like Python objects with arbitrary levels of nesting. What is a container can be re-defined for each application. By default, lists, tuples and dicts are considered containers and everything else is a leaf. Then the following are examples of pytrees: ```python [1, "a", np.arange(3)] # 3 leaves [1, {"k1": 2, "k2": (3, 4)}, 5] # 5 leaves np.arange(5) # 1 leaf ``` What makes pytrees so powerful are the operations defined for them. The most important ones are: - `tree_flatten`: Convert any pytree into a flat list of leaves + metadata - `tree_unflatten`: The inverse of `tree_flatten` - `tree_map`: Apply a function to all leaves in a pytree - `leaf_names`: Generate a list of names for all leaves in a pytree The above examples of pytrees would look as follows when flattened (with a default definition of containers): ```python [1, "a", np.arange(3)] [1, 2, 3, 4, 5] [np.arange(5)] ``` By adding numpy arrays to the registry of container like objects, each of the three examples above would have five leafs. The flattened versions would look as follows: ```python [1, "a", 0, 1, 2] [1, 2, 3, 4, 5] [0, 1, 2, 3, 4] ``` Needless to say, it is possible to register anything as container. For example, we would add pandas.Series and pandas.DataFrame (with varying definitions, depending on the application). ## Difference between pytrees in JAX and estimagic Most JAX functions [only work with Pytrees of arrays](https://jax.readthedocs.io/en/latest/pytrees.html#pytrees-and-jax-functions) and scalars, i.e. pytrees where container types are dicts, lists and tuples and all leaves are arrays or scalars. We will just call them pytrees of arrays because scalars are converted to arrays by JAX. There are two ways to look at such pytrees: 1. As pytree of arrays -> `tree_flatten` produces a list of arrays 1. As pytree of numbers -> `tree_flatten` produces a list of numbers The only difference between the two perspectives is that for the second one, arrays have been registered as container types that can be flattened. In JAX the term `ravel` instead of `flatten` is sometimes used to make clear that the second perspective is meant. Estimagic functions work with slightly more general pytrees. On top of arrays, they can also contain scalars, pandas.Series and pandas.DataFrames. Again, there are two possible ways to look at such pytrees: 1. As pytree of arrays, numbers, Series and DataFrames -> `tree_flatten` produces a list of arrays numbers, Series and DataFrames. 1. As pytree of numbers -> `tree_flatten` produces a list of numbers Again, the difference between the two is which objects are registered as container types and the rules to flatten and unflatten them are defined. While numpy arrays, scalars and pandas.Series have only one natural way of defining the flattening rules, this becomes more complex for DataFrames due to the way `params` DataFrames were used in estimagic before. We define the following rules: If a DataFrame contains a column called `"value"`, we interpret them as classical estimagic DataFrame and only consider the entries in the `"value"` column when flattening the DataFrame into a list of numbers. If there is no column `"value"`, all numeric columns of the DataFrame are considered. Note that internally, we will sometimes define flattening rules such that only some other columnn, e.g. only `"lower_bound"` is considered. However we never look at more than one column of a classical estimagic params DataFrame at a time. To distinguish between the different pytrees we use the terms JAX-pytree and estimagic-pytree. ## Optimization with pytrees In this section we look at possible ways to specify optimizations when parameters and some outputs of criterion functions can be estimagic-pytrees. As an example we use a hypothetical criterion function with pytree inputs and outputs to describe how a user can optimize it. We also give a rough intuition what happens behind the scenes. ### The criterion function Consider a criterion function that takes parameters in the following format: ```python params = { "delta": 0.95, "utility": pd.DataFrame( [[0.5, 0]] * 3, index=["a", "b", "c"], columns=["value", "lower_bound"] ), "probs": np.array([[0.8, 0.2], [0.3, 0.7]]), } ``` The criterion function returns a dictionary of the form: ```python { "value": 1.1, "contributions": {"a": np.array([0.36, 0.25]), "b": 0.49}, "root_contributions": {"a": np.array([0.6, 0.5]), "b": 0.7}, } ``` ### Run an optimization ```python from estimagic import minimize minimize( criterion=crit, params=params, algorithm="scipy_lbfgsb", ) ``` The internal optimizer (in this case the lbfgsb algorithm from scipy) will see a wrapped version of `crit`. That version takes a 1d numpy array as its only argument and returns a scalar float (the `"value"` entry of the result of `crit`). Numerical derivatives are also taken on that function. If instead a derivative based least squares optimizer like `"scipy_ls_dogbox"` had been used, the internal optimizer would see a modified version of `crit` that takes a 1d numpy array and returns a 1d numpy array (the flattened version of the `"root_contributions"` entry of the result of `crit`). ### The optimization output The following entries of the output of minimize are affected by the change: - `"solution_params"`: A pytree with the same structure as `params` - `"solution_criterion"`: The output dictionary of `crit` evaluated solution params - `solution_derivative`: Maybe we should not even have this entry. ```{note} We need to discuss if and in which form we want to have a solution derivative entry. In it's current form it is useless if constraints are used. This gets worse when we allow for pytrees and translating this into a meaningful shape might be very difficult. ``` ### Add bounds Bounds on parameters that are inside a DataFrame with `"value"` column can simply be specified as before. For all others, there are separate `lower_bounds` and `upper_bounds` arguments in `maximize` and `minimize`. `lower_bounds` and `upper_bounds` are pytrees of the same structure as `params` or a subtree that preserves enough structure to match all bounds. For example: ```python minimize( criterion=crit, params=params, algorithm="scipy_lbfgsb", lower_bounds={"delta": 0}, upper_bounds={"delta": 1}, ) ``` This would add bounds for delta, keep the bounds on all `"utility"` parameters, and leave the `"probs"` parameters unbounded. ### Add a constraint Currently, parameters to which a constraint is applied are selected via a `"loc"` or `"query"` entry in the constraints dictionary. This keeps working as long as params are specified as a single DataFrame containing a `"value"` column. If a more general pytree is used we need a "selector" entry instead. The value of that entry is a callable that takes the pytree and returns selected parameters. The `selector` function may return the parameters in the form of an estimagic-pytree. Should order play a role for the constraints (e.g., increasing) the constraint will be applied to the flattened version of the pytree returned by the `selector` function. However, in the case that order matters, we advise users to return one-dimensional arrays (explicit is better than implicit). As an example, let's add probability constraints for each row of `"probs"`: ```python constraints = [ {"selector": lambda params: params["probs"][0], "type": "probability"}, {"selector": lambda params: params["probs"][1], "type": "probability"}, ] minimize( criterion=crit, params=params, algorithm="scipy_lbfgsb", constraints=constraints, ) ``` The required changes to support this are relatively simple. This is because most functions that deal with constraints already work with a 1d array of parameters and the `"loc"` and `"query"` entries of constraints are internally translated to positions in that array very early on. ### Derivatives during optimization If numerical derivatives are used, they are already taken on a modified function that maps from 1d numpy array to scalars or 1d numpy arrays. Allowing for estimagic-pytrees in parameters and criterion outputs will not pose any difficulties here. Closed form derivatives need to have the following interface: They expect `params` in the exact same format as the criterion function as first argument. They return a derivative in the same format as our numerical derivative functions or JAXs autodiff functions when applied to the criterion function. ## Numerical derivatives with pytrees ### Problem: Higher dimensional extensions of pytrees The derivative of a function that maps from a 1d array to a 1d array (usually called Jacobian) is a 2d matrix. If the 1d arrays are replaced by pytrees, we need a two dimensional extension of the pytrees. Below we will look at how JAX does this and why we cannot simply copy that solution. ### The JAX solution Let's look at an example. We first define a function in terms of 1d arrays and then in terms of pytrees and look at a JAX calculated jacobian in both cases: ```python def square(x): return x**2 x = jnp.array([1, 2, 3, 4, 5, 6.0]) jacobian(square)(x) ``` ```bash DeviceArray([[ 2., 0., 0., 0., 0., 0], [ 0., 4., 0., 0., 0., 0], [ 0., 0., 6., 0., 0., 0], [ 0., 0., 0., 8., 0., 0], [ 0., 0., 0., 0., 10., 0], [ 0., 0., 0., 0., 0., 12]], dtype=float32) ``` ```python def tree_square(x): out = { "c": x["a"] ** 2, "d": x["b"].flatten() ** 2, } return out tree_x = {"a": jnp.array([1, 2.0]), "b": jnp.array([[3, 4], [5, 6.0]])} jacobian(tree_square)(tree_x) ``` Instead of showing the entire results, let's just look at the resulting tree structure and array shapes: ```python { "c": { "a": (2, 2), "b": (2, 2, 2), }, "d": { "a": (4, 2), "b": (4, 2, 2), }, } ``` The outputs for hessians have even deeper nesting and three dimensional arrays inside the nested dictionary. Similarly, we would get higher dimensional arrays if one of the original pytrees had already contained a 2d array. ### Extending the JAX solution to estimagic-pytrees JAX pytrees can only contain arrays, whereas estimagic-pytrees may contain scalars, pandas.Series and pandas.DataFrames (with or without `"value"` column). Unfortunately, this poses non-trivial challenges for numerical derivatives because those data types have no natural extension in arbtirary dimensions. Our solution needs to fulfill two requirements: 1\. Compatible with JAX in the sense than whenever a derivative can be calculated with JAX it can also be calculated with estimagic and the result has the same structure. 2. Compatible with the rest of estimagic in the sense that any function that can be optimized can also be differentiated. In the special case of differentiating with respect to a DataFrame it also needs to be backwards compatible. A solution that achieves this is to treat Series and DataFrames with `"value"` columns as 1d arrays and other DataFrames as 2d arrays, then proceed as in JAX and finally try to preserve as much index and column information as possible. This leads to very natural results in the typical usecases with flat dicts of Series or params DataFrames both as inputs and outputs and is backwards compatible with everything that is supported already. However, similar to JAX, not everything that is supported will also be a good idea. Predicting where a pandas Object is preserved and where it will be replaced by an array might be hard for very nested pytrees. However, these rules are mainly defined to avoid hard limitations that have to be checked and documented. Users will learn to avoid too much complexity by avoiding complex pytrees as inputs and outputs at the same time. To see this in action, let's look at an example. We repeat the example from the JAX interface above with the following changes: 1. The 1d numpy array in x["a"] is replaced by a DataFrame with `"value"` column 1. The "d" entry in the output becomes a Series instead of a 1d numpy array. ```python def pd_tree_square(x): out = { "c": x["a"]["value"] ** 2, "d": pd.Series(x["b"].flatten() ** 2, index=list("jklm")), } return out pd_tree_x = { "a": pd.DataFrame(data=[[1], [2]], index=["alpha", "beta"], columns=["value"]), "b": np.array([[3, 4], [5, 6]]), } pd_tree_square(pd_tree_x) ``` ``` { 'c': "alpha" 1 "beta" 4 dtype: int64, 'd': "j" 9 "k" 16 "l" 25 "m" 36 dtype: int64, } ``` The resulting shapes of the jacobian will be the same as before. For all arrays with only two dimensions we can preserve some information from the Series and DataFrame indices. On the higher dimensional ones, this will be lost. ```python { "c": { "a": (2, 2), # df with columns ["alpha", "beta"], index ["alpha", "beta"] "b": (2, 2, 2), # numpy array without label information }, "d": { "a": (4, 2), # columns ["alpha", "beta"], index [0, 1, 2, 3] "b": (4, 2, 2), # numpy array without label information }, } ``` To get more intuition for the structure of the result, let's add a few labels to the very first jacobian: ```{eval-rst} +--------+----------+----------+----------+----------+----------+----------+----------+ | | | a | | b | | | | +--------+----------+----------+----------+----------+----------+----------+----------+ | | | alpha | beta | j | k | l | m | +--------+----------+----------+----------+----------+----------+----------+----------+ | c | alpha | 2 | 0 | 0 | 0 | 0 | 0 | + +----------+----------+----------+----------+----------+----------+----------+ | | beta | 0 | 4 | 0 | 0 | 0 | 0 | +--------+----------+----------+----------+----------+----------+----------+----------+ | d | 0 | 0 | 0 | 6 | 0 | 0 | 0 | + +----------+----------+----------+----------+----------+----------+----------+ | | 1 | 0 | 0 | 0 | 8 | 0 | 0 | + +----------+----------+----------+----------+----------+----------+----------+ | | 2 | 0 | 0 | 0 | 0 | 10 | 0 | + +----------+----------+----------+----------+----------+----------+----------+ | | 3 | 0 | 0 | 0 | 0 | 0 | 12 | +--------+----------+----------+----------+----------+----------+----------+----------+ ``` The indices ["j", "k", "l", "m"] unfortunately never made it into the result because they were only applied to elements that already came from a 2d array and thus always have a 3d Jacobian, i.e. the result entry `["c"][b"]` is a reshaped version of the upper right 2 by 4 array and the result entry `["d"]["b"]` is a reshaped version of the lower right 4 by 4 array. ### Implementation We use the following terminology to describe the implementation: - input_tree: The pytree containing parameters, i.e. inputs to the function that is differentiated. - output_tree: The pytree that is returned by the function being differentiated - derivative_tree: The pytree we want to generate, i.e. the pytree that would be returned by JAX jacobian. - flat_derivative: The matrix version of the derivative_tree To simply reproduce the JAX behavior with pytrees of arrays, we could proceed in the following steps: - Create a modified function that maps from 1d array to 1d array - Calculate flat_derivative by taking numerical derivatives just as before - Calculate the shapes of all arrays in derivative_tree by concatenating the shapes of the cartesian product of flattend output_tree and input_tree - Calculate the 2d versions of those arrays by taking the product over elements in the shape tuple before concatenating. - Create a list of lists containing all arrays that will be in derivative_tree. The values are taken from flat_derivative, using the previously calculated shapes. - call `tree_unflatten` on the inner lists with the treedef corresponding to input_tree. - call `tree_unflatten` on the result of that with the treedef corresponding to output_tree. To implement the extension to estimagic pytrees we would probably do exactly the same but have a bit more preparation and post-processing to do. ## General aspects of pytrees in estimation functions ### Estimation summaries Currently, estimation summaries are DataFrames. The estimated parameters are in the `"value"` column. There are other columns with standard errors, p-values, significance stars and confidence intervals. This is another form of higher dimensional extension of pytrees, where we need to add additional columns. There are two ways in which estimation summaries could be presented. I suggest we offer both. The first is more geared towards generating estimation tables and serving as actual summary to be looked at in a jupyter notebook. It is also backwards compatible and should thus be the default. The second is more geared towards further calculations. There will be utility functions to convert between the two. Both formats will be explained using the `params` pytree from the optimization example (reproduced here for convenience): #### Format 1: Everything becomes a DataFrame In this approach we do the following conversions: 1. numpy arrays are flattened and converted to DataFrames with one column called `"value"`. The index contains the original positions of elements. 1. pandas.Series are converted to DataFrames. The index remains unchanged. The column is called `"value"`. 1. scalars become DataFrames with one row with index 0 and one column called `"value"`. 1. DataFrames without `"value"` column are stacked into a DataFrame with just one column called `"value"`. 1. DataFrames with `"value"` column are reduced to that column. After these transformations, all numbers of the original pytree are stored in DataFrames with `"value"` column. Additional columns with standard errors and the like can then simply be assigned as before. For more intuition, let's see how this would look in an example. For simplicity we only add a column with stars and ommit standard errors, p-values and confidence intervals. We use the same example as in the optimization section: ```python params = { "delta": 0.95, "utility": pd.DataFrame( [[0.5, 0]] * 3, index=["a", "b", "c"], columns=["value", "lower_bound"] ), "probs": np.array([[0.8, 0.2], [0.3, 0.7]]), } ``` ``` { 'delta': value stars 0 0.95 ***, 'utility': value stars a 0.5 ** b 0.5 ** c 0.5 **, 'probs': value stars 0 0 0.8 *** 1 0.2 * 1 0 0.3 ** 1 0.7 ***, } ``` #### Format 2: Dictionary of pytrees The second solution is a dictionary of pytrees the keys are the columns of the current summary but probably in plural, i.e. "values", "standard_errors", "p-values", ...; Each value is a pytree with the exact same structure as `params`. If this pytree contains DataFrames with `"value"` column, only that column is updated. i.e. standard errors would be accessed via `summary["standard_errors"]["my_df"]["value"]`. ### Representation of covariance matrices A covariance matrix is a two dimensional extension of a `params` pytree. We could theoretically handle it exactly the same way as Jacobians. However, this would not be useful for statistical tests and visualization if it contains more than 2 dimensional arrays (as the Jacobian example does). We thus propose to have two possible formats in which covariance matrices can be returned: 1. The pytree variant described in the above Jacobian example. This will be useful to look at sub-matrices of the full covariance matrix as long as the `params` pytree only contains one dimensional arrays, Series and DataFrames with `"value"` columns. 1. A DataFrame containing the covariance matrix of the flattened parameter vector. The index and columns of the DataFrames can be constructed from the `leaf_names` function in `pybaum`. We could also triviall add a function there that constructs an index that is easier to work with for selecting elements and let the user choose between the two versions. The function that maps from the flat version (which would be calculated internally) to the pytree version is the same as we need for numerical derivatives. The inverse of that function is probably not too difficult to implement and can also be useful for derivatives. ### params Everything that can be used as `params` in optimization and differentiation can also be used as `params` in estimation. The registries used in pytree functions are identical. ## ML specific aspects of pytrees The output of the log likelihood functions is a dictionary with the entries: - `"value"`: a scalar float - `"contributions"`: a 1d numpy array or pandas.Series Moreover, there can be arbitrary additional entries. The only change is that `"contributions"` can now be any estimagic pytree. ## MSM specific aspects of pytrees ### Valid formats of empirical and simulated moments There are three types of moments in MSM estimation: - `empirical moments` - The output of `simulate_moments` - The output of `calculate_moments`, needed to get a moments covariance matrix. We propose that moments can be stored as any valid estimagic pytree but of course all three types of moments have to be aligned, i.e. be stored in a tree of the same structure. We will raise an error if the trees do not have the same structure. This is a generalization of an interface that has already proven useful in [respy](https://github.com/OpenSourceEconomics/respy), [sid](https://github.com/covid-19-impact-lab/sid) and other applications. In the future, the project specific implementations of flatten and unflatten functions could simply be deleted. ### Representation of the weighting matrix and moments_cov The weighting matrix for MSM estimation is represented as a DataFrame in the same way as the flat representation of the covariance matrices. Of course, the conversion functions that work for covariance matrices would also work here, but it is highly unlikely that a different representation of a weighting matrix is ever needed. Note that the user does not have to construct this weighting matrix manually. They can generate them using `get_moments_cov` and `get_weighting_matrix`, so they do not need any knowledge of how the flattening works. ### Pepresentation of sensitivity measures Sensitivity measures are similar to covariance matrices in the sense that they require a two dimensional extension of pytrees. The only difference is that for covariance matrices the two pytrees the same (namely the `params`) and for sensitivity measures they are different (one is `params`, the other `moments`). We therefore suggest to use the same solution, i.e. to offer a flat representation in form of a DataFrame, a pytree representation and functions to convert between the two. ## Compatibility with estimation tables Estimation tables are constructed from estimation summaries. This continues to work for summaries where everything has been converted to DataFrames. Users will select individual DataFrames from a pytree of DataFrames, possibly concatenate or filter them and pass them to the estimation table function. ## Compatibility with plotting functions The following functions are affected: - `plot_univariate_effects` - `convergence_plot` - `lollipop_plot` - `derivative_plot` Most of them can be adjusted easily to the proposed changes. On all others we will simply raise errors and provide tutorials to work around the limitations. ## Compatibility with Dashboard The main challenge for the dashboard is that pytrees have no natural multi-column extension and thus it becomes harder to specify a group or name column. However, these features have not been used very much anyways. We propose to write a better automatic grouping and naming function for pytrees. That way it is simply not necessary to provide group and name columns and most of the users will get a better dashboard experience. Rules of thumb for both should be: 1. Only parameters where the start values have a similar magnitude can be in the same group, i.e. displayed in one lineplot. 1. Parameters that are close to each other in the tree (i.e. have a common beginning in their leaf_name should be in the same group. 1. The plot title should subsume the commen parts of the tree-structure (i.e. name we get from `pybaum.leaf_names`. 1. Most line plots should have approximately 5 lines, none should have more than 8. ## Advanced options for functions that work with pytrees There are two argument to `tree_flatten` and other pytree functions that determine which entries in a pytree are considered a leaf and which a container as well as how containers are flattened. 1. `registry` and 2. `is_leaf`. See the documentation of `pybaum` for details. To allow for absolute flexibility, each function that works with pytrees needs to allow a user to pass in a `registry` and an `is_leaf` argument. If a function works with multiple pytrees (e.g. in `estimate_msm` the `params` are a pytree and `emprirical_moments` are a pytree) it needs to allow users to pass in multiple registries and is_leaf functions (e.g. `params_registry`, `params_is_leaf` and `moments_registry`, `moments_is_leaf`. However, we need only as many registries as there are different pytrees. For example since `simulated_moments` and `empirical_moments` always need to be pytrees with the same structure, they do not need separate registries and is_leaf functions. ## Pytree related reasons for a switch to result objects There will be an other EEP that proposes to replace the result dictionaries we currently use everywhere in estimagic by result objects. While this in not completely related to pytrees, the switch to pytrees provides a few additional reasons: 1. Since we sometimes provide provide results in several formats (e.g. summaries as dict of pytrees and as pytree of DataFrames), the result dictionary would become too large and confusing. Having result objects that just calculate specific formats on demand can alleviate this. 1. The result object can serve as a simplfied wrapper to pytree functions and pytree conversion functions between pytree formats that abstracts from registry, is_leaf and treedefs. 1. Results objects allow to define a `__repr__` which becomes really useful as soon as parameters are not just one DataFrame but for example, a dict of DataFrames. ## Compatibility with JAX autodiff While we allow for pytrees of arrays, numbers and DataFrames, JAX only allows pytrees of arrays and numbers for automatic differentiation. If you want to use automatic differentiation with estimagic you will thus have to restrict yourself in the way you specify parameters. [pybaum]: https://github.com/OpenSourceEconomics/pybaum (eeptyping)= # EP-02: Static typing ```{eval-rst} +------------+------------------------------------------------------------------+ | Author | `Janos Gabler `_ | +------------+------------------------------------------------------------------+ | Status | Accepted | +------------+------------------------------------------------------------------+ | Type | Standards Track | +------------+------------------------------------------------------------------+ | Created | 2024-05-02 | +------------+------------------------------------------------------------------+ | Resolution | | +------------+------------------------------------------------------------------+ ``` ## Abstract This enhancement proposal explains the adoption of static typing in optimagic. The goal is to reap a number of benefits: - Users will benefit from IDE tools such as easier discoverability of options and autocompletion. - Developers and users will find code easier to read due to type hints. - The codebase will become more robust due to static type checking and use of stricter types in internal functions. Achieving these goals requires more than adding type hints. optimagic is currently mostly [stringly typed](https://wiki.c2.com/?StringlyTyped). For example, optimization algorithms are selected via strings. Another example are [constraints](https://estimagic.readthedocs.io/en/latest/how_to_guides/optimization/how_to_specify_constraints.html), which are dictionaries with a fixed set of required keys. This enhancement proposal outlines how we can accommodate the changes needed to reap the benefits of static typing without breaking users' code in too many places. ## Motivation and resources - [Writing Python like it's Rust](https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html). A very good blogpost that summarizes the drawbacks of "stringly-typed" Python code and shows how to incorporate typing philosophies from Rust into Python projects. Read this if you don't have time to read the other resources. - [Robust Python](https://www.oreilly.com/library/view/robust-python/9781098100650/), an excellent book that discusses how to design code around types and provides an introduction to static type checkers in Python. - [jax enhancement proposal](https://jax.readthedocs.io/en/latest/jep/12049-type-annotations.html) for adopting static typing. It has a very good discussion on benefits of static typing. - [Subclassing in Python Redux](https://hynek.me/articles/python-subclassing-redux/) explains which types of subclassing are considered harmful and was very helpful for designing this proposal. (design-philosophy)= ## Design Philosophy The core principles behind this enhancement proposal can be summarized by the following points. This is an extension to our existing [styleguide](https://estimagic.org/en/latest/development/styleguide.html) which will be updated if this proposal is accepted. - User facing functions should be generous regarding their input type. Example: the `algorithm` argument can be a string, `Algorithm` class or `Algorithm` instance. The `algo_options` can be an `AlgorithmOptions` object or a dictionary of keyword arguments. - User facing functions should be strict about their output types. A strict output type does not just mean that the output type is known (and not a generous Union), but that it is a proper type that enables static analysis for available attributes. Example: whenever possible, public functions should not return dicts but proper result types (e.g. `OptimizeResult`, `NumdiffResult`, ...) - Internal functions should be strict about input and output types; Typically, a public function will check all arguments, convert them to a proper type and then call an internal function. Example: `minimize` will convert any valid value for `algorithm` into an `Algorithm` instance and then call an internal function with that type. - Each argument that previously accepted strings or option dictionaries now also accepts input types that are more amenable to static analysis and offer better autocomplete. Example: `algo_options` could just be a dict of keyword arguments. Now it can also be an `AlgorithmOptions` instance that enables autocomplete and static analysis for attribute access. - Fixed field types should only be used if all fields are known. An example where this is not the case are collections of benchmark problems, where the set of fields depends on the selected benchmark sets and other things. In such situations, dictionaries that map strings to BenchmarkProblem objects are a good idea. - For backwards compatibility and compatibility with SciPy, we allow things we don't find ideal (e.g. selecting algorithms via strings). However, the documentation should mostly show our prefered way of doing things. Alternatives can be hidden in tabs and expandable boxes. - Whenever possible, use immutable types. Whenever things need to be changeable, consider using an immutable type with copy constructors for modified instances. Example: instances of `Algorithm` are immutable but using `Algorithm.with_option` users can create modified copies. - The main entry point to optimagic are functions, objects are mostly used for configuration and return types. This takes the best of both worlds: we get the safety and static analysis that (in Python) can only be achieved using objects but the beginner friendliness and freedom provided by functions. Example: Having a `minimize` function, it is very easy to add the possibility of running minimizations with multiple algorithms in parallel and returning the best value. Having a `.solve` method on an algorithm object would require a whole new interface for this. ## Changes for optimization The following changes apply to all functions that are directly related to optimization, i.e. `maximize`, `minimize`, `slice_plot`, `criterion_plot`, `params_plot`, `count_free_params`, `check_constraints` and `OptimizeResult`. ### The objective function #### Current situation The objective or criterion function is the function being optimized. The same criterion function can work for scalar, least-squares and likelihood optimizers. Moreover, a criterion function can return additional data that is stored in the log file (if logging is active). All of this is achieved by returning a dictionary instead of just a scalar float. For the simplest case, where only scalar optimizers are used, `criterion` returns a float. Here are two examples of this simple case. The **first example** represents `params` as a flat numpy array and returns a float. This would also be compatible with SciPy: ```python def sphere(params: np.ndarray) -> float: return params @ params ``` The **second example** also returns a float but uses a different format for the parameters: ```python def dict_sphere(params: dict) -> float: return params["a"] ** 2 + params["b"] ** 2 ``` If the user wants the criterion function to be compatible with specialized optimizers for least-squares problems, the criterion function needs to return a dictionary. ```python def least_squares_sphere(params: np.ndarray) -> dict[str, Any]: return {"root_contributions": params} ``` Here the `"root_contributions"` are the least-squares residuals. The dictionary key tells optimagic how to interpret the output. This is needed because optimagic has no way of finding out whether a criterion function that returns a vector (or pytree) is a least-squares function or a likelihood function. Of course all specialized problems can still be solved with scalar optimizers. The criterion function can also return a dictionary, if the user wants to store some information in the log file. This is independent of having a least-squares function or not. An example is: ```python def logging_sphere(x: np.ndarray) -> dict[str, Any]: return {"value": x @ x, "mean": x.mean(), "std": x.std()} ``` Here `"value"` is the actual scalar criterion value. All other fields are unknown to optimagic and therefore just logged in the database if logging is active. The specification of likelihood functions is very analogous to least-squares functions and therefore omitted here. **Things we want to keep** - Allow using the same criterion function for scalar, likelihood and least-squares optimizers. This feature makes it easy to try out and compare very different algorithms with minimal code changes. - No restrictions on the type of additional arguments of the criterion function. - Maintain compatibility with scipy.optimize when the criterion function returns a scalar. **Problems** - Most users of optimagic find it hard to write criterion functions that return the correct dictionary. Therefore, they don't use the logging feature and we often get questions about specifying least-squares problems correctly. - Internally we can make almost no assumptions about the output of a criterion function, making the code that processes the criterion output very complex and full of if conditions. - We only know whether the specified criterion function is compatible with the selected optimizer after we evaluate it once. This means that users see errors only very late. - While optional, in least-squares problems it is possible that a user specifies `root_contributions`, `contributions` and `value` even though any of them could be constructed out of the `root_contributions`. This redundancy of information means that we need to check the consistency of all user provided function outputs. #### Proposal In the current situation, the dictionary return type solves two different problems that will now be solved separately. ##### Specifying different problem types The simplest way of specifying a least-squares function becomes: ```python import optimagic as om @om.mark.least_squares def ls_sphere(params): return params ``` Analogously, the simplest way of specifying a likelihood function becomes: ```python @om.mark.likelihood def ll_sphere(params): return params**2 ``` The simplest way of specifying a scalar function stays unchanged, but optionally a `mark.scalar` decorator can be used: ```python @om.mark.scalar # this is optional def sphere(params): return params @ params ``` Except for the decorators, these three functions are specified the same way as in other python libraries that support specialized optimizers (e.g. `scipy.optimize.least_squares`). The reason why we need the decorators is that we support all kinds of optimizers in the same interface. ##### Return additional information If users additionally want to return information that should be stored in the log file, they need to use a specific Object as return type. ```python @dataclass(frozen=True) class FunctionValue: value: float | PyTree info: dict[str, Any] ``` An example of a least-squares function that also returns additional info for the log file would look like this: ```python from optimagic import FunctionValue @om.mark.least_squares def least_squares_sphere(params): out = FunctionValue( value=params, info={"p_mean": params.mean, "p_std": params.std()} ) return out ``` And analogous for scalar and likelihood functions, where again the `mark.scalar` decorator is optional. ##### Optionally replace decorators by type hints The purpose of the decorators is to tell us the output type of the criterion function. This is necessary because there is no way of distinguishing between likelihood and least-squares functions from the output alone and because we want to know the function type before we evaluate the function once. An alternative that might be more convenient for advanced Python programmers would be to do this via type hints. In this case, the return types need to be a bit more fine-grained: ```python @dataclass(frozen=True) class ScalarFunctionValue(FunctionValue): value: float info: dict[str, Any] @dataclass(frozen=True) class LeastSquaresFunctionValue(FunctionValue): value: PyTree info: dict[str, Any] @dataclass(frozen=True) class LikelihoodFunctionValue(FunctionValue): value: PyTree info: dict[str, Any] ``` A least-squares function could then be specified without decorator as follows: ```python from optimagic import LeastSquaresFunctionValue def least_squares_sphere(params: np.ndarray) -> LeastSquaresFunctionValue: out = LeastSquaresFunctionValue( value=params, info={"p_mean": params.mean, "p_std": params.std()} ) return out ``` This approach works nicely in projects that use type hints already. However, it would be hard for users who have never heard about type hints. Therefore, we should implement it but not use it in beginner tutorials and always make clear that this is completely optional. ##### Summary of output types The output type of the objective function is `float | PyTree[float] | FunctionValue`. ### Bundling bounds #### Current situation Currently we have four arguments of `maximize`, `minimize`, and related functions that let the user specify bounds: ```python om.minimize( # ... lower_bounds=params - 1, upper_bounds=params + 1, soft_lower_bounds=params - 2, soft_upper_bounds=params + 2, # ... ) ``` Each of them is a pytree that mirrors the structure of `params` or `None` **Problems** - Usually, all of these arguments are used together and passing them around individually is annoying. - The names are very long because the word `bounds` is repeated. #### Proposal We bundle the bounds together in a `Bounds` type: ```python bounds = om.Bounds( lower=params - 1, upper=params + 1, soft_lower=params - 2, soft_upper=params + 2, ) om.minimize( # ... bounds=bounds, # ... ) ``` As a bonus feature, the `Bounds` type can do some checks on the bounds at instance creation time such that users get errors before running an optimization. Using the old arguments will be deprecated. Since there is no need to modify instances of `Bounds`, it should be immutable. To improve the alignment with SciPy, we can also allow users to pass a `scipy.optimize.Bounds` object as bounds. Internally, this will be converted to our `Bounds` object. ### Constraints #### Current situation Currently, constraints are dictionaries with a set of required keys. The exact requirements depend on the type of constraints and even on the structure of `params`. Each constraint needs a way to select the parameters to which the constraint applies. There are three dictionary keys for this: - `"loc"`, which works if params are numpy arrays, `pandas.Series` or `pandas.DataFrame`. - `"query"`, which works only if `params` are `pandas.DataFrame` - `"Selector"`, which works for all valid formats of `params`. Moreover, each constraint needs to specify its type using the `"type"` key. Some constraints have additional required keys: - Linear constraints have `"weights"`, `"lower_bound"`, `"upper_bound"`, and `"value"`. - Nonlinear constraints have `"func"`, `"lower_bound"`, `"upper_bound"`, and `"value"`. Details and examples can be found [here](https://estimagic.readthedocs.io/en/latest/how_to_guides/optimization/how_to_specify_constraints.html). **Things we want to keep** - The constraints interface is very declarative; Constraints purely collect information and are completely separate from the implementation. - All three ways of selecting parameters have their strength and can be very concise and readable in specific applications. **Problems** - Constraints are hard to document and generally not understood by most users. - Having multiple ways of selecting parameters (not all compatible with all `params` formats) is confusing for users and annoying when processing constraints. We have to handle the case where no selection or multiple selections are specified. - Dicts with required keys are brittle and do not provide autocomplete. This is made worse by the fact that each type of constraint requires different sets of keys. #### Proposal 1. We implement simple dataclasses for each type of constraint. 1. We get rid of `loc` and `query` as parameter selection methods. Instead, we show in the documentation how both selection methods can be used inside a `selector` function. Examples of the new syntax are: ```python constraints = [ om.constraints.FixedConstraint(selector=lambda x: x[0, 5]), om.constraints.IncreasingConstraint(selector=lambda x: x[1:4]), ] res = om.minimize( fun=criterion, params=np.array([2.5, 1, 1, 1, 1, -2.5]), algorithm="scipy_lbfgsb", constraints=constraints, ) ``` Since there is no need to modify instances of constraints, they should be immutable. All constraints can subclass `Constraint` which will only have the `selector` attribute. During the deprecation phase, `Constraint` will also have `loc` and `query` attributes. The current `cov` and `sdcorr` constraints apply to flattened covariance matrices, as well as standard deviations and flattened correlation matrices. This comes from a time where optimagic only supported an essentially flat parameter format (`DataFrames` with `"value"` column). We can exploit the current deprecation cycle to rename the current `cov` and `sdcorr` constraints to `FlatCovConstraint` and `FlatSdcorrConstraint`. This prepares the introduction of a more natural `CovConstraint` and `SdcorrConstraint` later. (algorithm-selection)= ### Algorithm selection #### Current situation `algorithm` is a string or a callable that satisfies the internal algorithm interface. If the user passes a string, we look up the algorithm implementation in a dictionary containing all installed algorithms. We implement suggestions for typical typos based on fuzzy matching of strings. **Things we want to keep** - optimagic can be used just like scipy **Problems** - There is no autocomplete. - It is very easy to make typos and they only get caught at runtime. - Users cannot select algorithms without reading the documentation. #### Proposal The following proposal is quite ambitious and split into multiple steps. Thanks to [@schroedk](https://github.com/schroedk) for helpful discussions on this topic. ##### Step 1: Passing algorithm classes and objects For compatibility with SciPy we continue to allow algorithm strings. However, the preferred ways of selecting algorithms are now: 1. Passing an algorithm class 1. Passing a configured algorithm object Both new ways become possible because of changes to the internal algorithm interface. See [here](algorithm-interface) for the proposal. We remove the possibility of passing callables that comply with the old internal algorithm interface. In a simple example, algorithm selection via algorithm classes looks as follows: ```python om.minimize( lambda x: x @ x, params=np.arange(5), algorithm=om.algorithms.scipy_neldermead, ) ``` Passing a configured instance of an algorithm looks as follows: ```python om.minimize( lambda x: x @ x, params=np.arange(5), algorithm=om.algorithms.scipy_neldermead(adaptive=True), ) ``` ##### Step 2: Achieving autocomplete without too much typing There are many ways in which the above behavior could be achieved with full autocomplete support. For reasons that will become clear in the next section, we choose to represent `algorithms` as a dataclass. Alternatives are enums, `__init__` files, NamedTuples, etc. A prototype for that dataclass looks as follows: ```python from typing import Type @dataclass(frozen=True) class Algorithms: scipy_neldermead: Type[ScipyNelderMead] = ScipyNelderMead scipy_lbfgsb: Type[ScipyLBFGSB] = ScipyLBFGSB # ... # many more # ... algorithms = Algorithms() ``` Currently, all algorithms are collected in a dictionary that is created programmatically. Representing algorithms in a static data structure instead requires a lot more typing and therefore code to maintain. This situation will become even worse with some of the features we propose below. Therefore, we want to automate the creation of the dataclass. To this end, we can write a function that automatically creates the code for the `Algorithms` dataclass. This function can be executed in a local pre-commit hook to make sure all generated code is up-to-date in every commit. It can also be executed in a [pytest hook](https://docs.pytest.org/en/7.1.x/how-to/writing_hook_functions.html) (before the collection phase) to make sure everything is up-to-date when tests run. Users of optimagic (and their IDEs) will never know that this code was not typed in by a human, which guarantees that autocomplete and static analysis will work without problems. ```{note} We can also use [pytest-hooks](https://docs.pytest.org/en/7.1.x/how-to/writing_hook_functions.html) to make sure the ``` ##### Step 3: Filtered autocomplete Having the flat `Algorithms` data structure would be enough if every user knew exactly which algorithm they want to use and just needed help typing in the name. However, this is very far from realistic. Most users have little knowledge about optimization algorithms. In the best case, they know a few properties of their problems (e.g. whether it is differentiable) and their goal (e.g. do they need a local or global solution). To exemplify what we want to achieve, assume a simplified situation with 4 algorithms. We only consider whether an algorithm is gradient free or gradient based. Here is the fictitious list: - `neldermead`: `gradient_free` - `bobyqa`: `gradient_free` - `lbfgs`: `gradient_based` - `slsqp`: `gradient_based` We want the following behavior: The user types `om.algorithms.` and autocomplete shows | | | --------------- | | `GradientBased` | | `GradientFree` | | `neldermead` | | `bobyqa` | | `lbfgs` | | `slsqp` | A user can either select one of the algorithms (lowercase) directly or filter further by selecting a category (CamelCase). This would look as follows: The user types `om.algorithms.GradientFree.` and autocomplete shows | | | ------------ | | `neldermead` | | `bobyqa` | Once the user arrives at an algorithm, a subclass of `Algorithm` is returned. This class will be passed to `minimize` or `maximize`. Passing configured instances of `Algorithm`s will be discussed in [Algorithm Options](algorithm-options). In practice, we would have a lot more algorithms and a lot more categories. Some categories might be mutually exclusive, in that case the second category is omitted after the first one is selected. We have the following categories: - `GradientBased` vs. `GradientFree` - `Local` vs. `Global` - `Bounded` vs. `Unbounded` - `Scalar` vs. `LeastSquares` vs. `Likelihood` - `LinearConstrained` vs. `NonlinearConstrained` vs. `Unconstrained` Potentially, we could also offer a `.All` attribute that returns a list of all currently selected algorithms. That way a user could for example loop over all `Bounded` and `GradientBased` `LeastSquares` algorithms and compare them in a criterion plot. These categories match nicely with our [algorithm selection tutorials](https://effective-programming-practices.vercel.app/scientific_computing/optimization_algorithms/objectives_materials.html). To achieve this behavior, we would have to implement something like this: ```python @dataclass(frozen=True) class GradientBasedAlgorithms: lbfgs: Type[LBFGS] = LBFGS slsqp: Type[SLSQP] = SLSQP @property def All(self) -> List[om.typing.Algorithm]: return [LBFGS, SLSQP] @dataclass(frozen=True) class GradientFreeAlgorithms: neldermead: Type[NelderMead] = NelderMead bobyqa: Type[Bobyqa] = Bobyqa @property def All(self) -> List[om.typing.Algorithm]: return [NelderMead, Bobyqa] @dataclass(frozen=True) class Algorithms: lbfgs: Type[LBFGS] = LBFGS slsqp: Type[SLSQP] = SLSQP neldermead: Type[NelderMead] = NelderMead bobyqa: Type[Bobyqa] = Bobyqa @property def GradientBased(self) -> GradientBasedAlgorithms: return GradientBasedAlgorithms() @property def GradientFree(self) -> GradientFreeAlgorithms: return GradientFreeAlgorithms() @property def All(self) -> List[om.typing.Algorithm]: return [LBFGS, SLSQP, NelderMead, Bobyqa] ``` If implemented by hand, this would require an enormous amount of typing and introduce a very high maintenance burden. Whenever a new algorithm was added to optimagic, we would have to register it in multiple nested dataclasses. The code generation approach detailed in the previous section can solve this problom. While it might have been overkill to achieve basic autocomplete, it is justified to achieve this filtering behavior. How the relevant information for filtering (e.g. whether an algorithm is gradient based) is collected, will be discussed in [internal algorithms](algorithm-interface). ```{note} The use of dataclasses is an implementation detail. This enhancement proposal only defines the autocomplete behavior we want to achieve. Everything else can be changed later as we see fit. ``` (algorithm-options)= ### Algorithm options Algorithm options refer to options that are not handled by optimagic but directly by the algorithms. Examples are convergence criteria, stopping criteria and advanced configuration of algorithms. Some of them are supported by many algorithms (e.g. stopping after a maximum number of function evaluations is reached), some are supported by certain classes of algorithms (e.g. most genetic algorithms have a population size, most trustregion algorithms allow to set an initial trustregion radius) and some of them are completely specific to one algorithm (e.g. ipopt has more than 100 very specific options, `nag_dfols` supports very specific restarting strategies, ...). While nothing can be changed about the fact that every algorithm supports different options (e.g. there is simply no trustregion radius in a genetic algorithm), we go very far in harmonizing `algo_options` across optimizers: 1. Options that are the same in spirit (e.g. stop after a specific number of iterations) get the same name across all optimizers wrapped in optimagic. Most of them even get the same default value. 1. Options that have non-descriptive (and often heavily abbreviated) names in their original implementation get more readable names, even if they appear only in a single algorithm. 1. Options that are specific to a well known optimizer (e.g. `ipopt`) are not renamed #### Current situation The user passes `algo_options` as a dictionary of keyword arguments. All options that are not supported by the selected algorithm are discarded with a warning. The names of most options are very descriptive (even though a bit too long at times). We implement basic namespaces by introducing a dot notation. Example: ```python options = { "stopping.max_iterations": 1000, "stopping.max_criterion_evaluations": 1500, "convergence.relative_criterion_tolerance": 1e-6, "convergence.scaled_gradient_tolerance": 1e-6, "initial_radius": 0.1, "population_size": 100, } ``` The option dictionary is then used as follows: ```python minimize( # ... algorithm="scipy_lbfgsb", algo_options=options, # ... ) ``` In the example, only the options `stopping.max_criterion_evaluations`, `stopping.max_iterations` and `convergence.relative_criterion_tolerance` are supported by `scipy_lbfgsb`. All other options would be ignored. ```{note} The `.` notation in `stopping.max_iterations` is just syntactic sugar. Internally, the option is called `stopping_max_iterations` because all options need to be valid Python variable names. ``` **Things we want to keep** - The ability to provide global options that are filtered for each optimizer. Mixing the options for all optimizers in a single dictionary and discarding options that do not apply to the selected optimizer allows to loop very efficiently over very different algorithms (without `if` conditions in the user's code). This is very good for quick experimentation, e.g. solving the same problem with three different optimizers and limiting each optimizer to 100 function evaluations. - The basic namespaces help to quickly see what is influenced by a specific option. This works especially well to distinguish stopping options and convergence criteria from other tuning parameters of the algorithms. However, it would be enough to keep them as a naming convention if we find it hard to support the `.` notation. - All options are documented in the optimagic documentation, i.e. we do not link to the docs of original packages. Now they will also be discoverable in an IDE. **Problems** - There is no autocomplete and the only way to find out which options are supported is the documentation. - A small typo in an option name can easily lead to the option being discarded. - Option dictionaries can grow very big. - The fact that option dictionaries are mutable can lead to errors, for example when a user wants to try out a grid of values for one tuning parameter while keeping all other options constant. #### Proposal We want to offer multiple entry points for passing additional options to algorithms. Users can pick the one that works best for their particular use-case. The current solution remains valid but not recommended. ##### Configured algorithms Instead of passing an `Algorithm` class (as described in [Algorithm Selection](algorithm-selection)) the user can create an instance of their selected algorithm. When creating the instance, they have autocompletion for all options supported by the selected algorithm. `Algorithm`s are immutable. ```python algo = om.algorithms.scipy_lbfgsb( stopping_max_iterations=1000, stopping_max_criterion_evaluations=1500, convergence_relative_criterion_tolerance=1e-6, ) minimize( # ... algorithm=algo, # ... ) ``` ##### Copy constructors on algorithms Given an instance of an `Algorithm`, a user can easily create a modified copy of that instance by using the `with_option` method. ```python # using copy constructors to create variants base_algo = om.algorithms.fides(stopping_max_iterations=1000) algorithms = [base_algo.with_option(initial_radius=r) for r in [0.1, 0.2, 0.5]] for algo in algorithms: minimize( # ... algorithm=algo, # ... ) ``` We can provide additional methods `with_stopping` and `with_convergence` that call `with_option` internally but provide two additional features: 1. They validate that the option is indeed a stopping/convergence criterion. 1. They allow to omit the `convergence_` or `stopping_` at the beginning of the option name and can thus reduce repetition in the option names. This recreates the namespaces we currently achieve with the dot notation: ```python # using copy constructors for better namespaces algo = ( om.algorithms.scipy_lbfgsb() .with_stopping( max_iterations=1000, max_criterion_evaluations=1500, ) .with_convergence( relative_criterion_tolerance=1e-6, ) ) minimize( # ... algorithm=algo, # ... ) ``` ##### Global option object As before, the user can pass a global set of options to `maximize` or `minimize`. We continue to support option dictionaries but also allow `AlgorithmOption` objects that enable better autocomplete and immutability. We can construct them using a similar pre-commit hook approach as discussed in [algorithm selection](algorithm-selection). Global options override the options that were directly passed to an optimizer. For consistency, `AlgorithmOptions` can offer the `with_stopping`, `with_convergence` and `with_option` copy-constructors, so users can modify options safely. Probably, this approach should be featured less prominently in the documentation as it offers no guarantees that the specified options are compatible with the selected algorithm. The previous example continues to work. Examples of the new possibilities are: ```python options = om.AlgorithmOptions( stopping_max_iterations=1000, stopping_max_criterion_evaluations=1500, convergence_relative_criterion_tolerance=1e-6, convergence_scaled_gradient_tolerance=1e-6, initial_radius=0.1, population_size=100, ) minimize( # ... algorithm=om.algorithms.scipy_lbfgsb, algo_options=options, # ... ) ``` ```{note} In my currently planned implementation, autocomplete will not work reliably for the copy constructors (`with_option`, `with_stopping` and `with_convergence`). The main reason is that most editors do not play well with `functools.wraps` or any other means of dynamic signature creation. For more details, see the discussions about the [Internal Algorithm Interface](algorithm-interface). ``` ### Custom derivatives Providing custom derivatives to optimagic is slightly complicated because we support scalar, likelihood and least-squares problems in the same interface. Moreover, we allow to either provide a `derivative` function or a joint `criterion_and_derivative` function that allow users to exploit synergies between evaluating the criterion and the derivative. #### Current situation The `derivative` argument can currently be one of three things: - A `callable`: This is assumed to be the relevant derivative of `criterion`. If a scalar optimizer is used, it is the gradient of the criterion value w.r.t. params. If a likelihood optimizer is used, it is the jacobian of the likelihood contributions w.r.t. params. If a least-squares optimizer is used, it is the jacobian of the residuals w.r.t. params. - A `dict`: The dict must have three keys `"value"`, `"contributions"` and `"root_contributions"`. The corresponding values are the three callables described above. - `None`: In this case, a numerical derivative is calculated. The `criterion_and_derivative` argument exactly mirrors `derivative` but each callable returns a tuple of the criterion value and the derivative instead. **Things we want to keep** - It is good that synergies between `criterion` and `derivative` can be exploited. - There are three arguments (`criterion`, `derivative`, `criterion_and_derivative`). This makes sure that every algorithm can run efficiently when looping over algorithms and keeping everything else equal. With SciPy's approach of setting `jac=True` if one wants to use a joint criterion and derivative function, a gradient free optimizer would have no chance of evaluating just the criterion. - Scalar, least-squares and likelihood problems are supported in one interface. **Problems** - A dict with required keys is brittle - Autodiff needs to be handled completely outside of optimagic - The names `criterion`, `derivative` and `criterion_and_derivative` are not aligned with scipy and very long. - Providing derivatives to optimagic is perceived as complicated and confusing. #### Proposal ```{note} The following section uses the new names `fun`, `jac` and `fun_and_jac` instead of `criterion`, `derivative` and `criterion_and_derivative`. ``` To improve the integration with modern automatic differentiation frameworks, `jac` or `fun_and_jac` can also be a string `"jax"` or a more autocomplete friendly enum `om.autodiff_backend.JAX`. This can be used to signal that the objective function is jax compatible and jax should be used to calculate its derivatives. In the long run we can add PyTorch support and more. Since this is mostly about a signal of compatibility, it would be enough to set one of the two arguments to `"jax"`, the other one can be left at `None`. Here is an example: ```python import jax.numpy as jnp import optimagic as om def jax_sphere(x): return jnp.dot(x, x) res = om.minimize( fun=jax_sphere, params=jnp.arange(5), algorithm=om.algorithms.scipy_lbfgsb, jac="jax", ) ``` If a custom callable is provided as `jac` or `fun_and_jac`, it needs to be decorated with `@om.mark.least_squares` or `om.mark.likelihood` if it is not the gradient of a scalar function values. Using the `om.mark.scalar` decorator is optional. For a simple least-squares problem this looks as follows: ```python import numpy as np @om.mark.least_squares def ls_sphere(params): return params @om.mark.least_squares def ls_sphere_jac(params): return np.eye(len(params)) res = om.minimize( fun=ls_sphere, params=np.arange(5), algorithm=om.algorithms.scipy_ls_lm, jac=ls_sphere_jac, ) ``` Note that here we have a least-squares problem and solve it with a least-squares optimizer. However, any least-squares problem can also be solved with scalar optimizers. While optimagic could convert the least-squares derivative to the gradient of the scalar function value, this is generally inefficient. Therefore, a user can provide multiple callables of the objective function in such a case, so we can pick the best one for the chosen optimizer. ```python @om.mark.scalar def sphere_grad(params): return 2 * params res = om.minimize( fun=ls_sphere, params=np.arange(5), algorithm=om.algorithms.scipy_lbfgsb, jac=[ls_sphere_jac, sphere_grad], ) ``` Since a scalar optimizer was chosen to solve the least-squares problem, optimagic would pick the `sphere_grad` as derivative. If a leas-squares solver was chosen, we would use `ls_sphere_jac`. ### Other option dictionaries #### Current situation We often allow to switch on some behavior with a bool or a string value and then configure the behavior with an option dictionary. Examples are: - `logging` (`str | pathlib.Path | False`) and `log_options` (dict) - `scaling` (`bool`) and `scaling_options` (dict) - `error_handling` (`Literal["raise", "continue"]`) and `error_penalty` (dict) - `multistart` (`bool`) and `multistart_options` Moreover we have option dictionaries whenever we have nested invocations of optimagic functions. Examples are: - `numdiff_options` in `minimize` and `maximize` - `optimize_options` in `estimate_msm` and `estimate_ml` **Things we want to keep** - Complex behavior like logging or multistart can be switched on in extremely simple ways, without importing anything and without looking up supported options. - The interfaces are very declarative and decoupled from our implementation. **Problems** - Option dictionaries are brittle and don't support autocomplete. - It can be confusing if someone provided `scaling_options` or `multistart_options` but they take no effect because `scaling` or `multistart` were not set to `True`. #### Proposal We want to keep a simple way of enabling complex behavior (with some default options) but get rid of having two separate arguments (one to switch the behavior on and one to configure it). This means that we have to be generous regarding input types. ##### Logging Currently we only implement logging via an sqlite database. All `log_options` are specific to this type of logging. However, logging is slow and we should support more types of logging. For this, we can implement a simple `Logger` abstraction. Advanced users could implement their own logger. After the changes, `logging` can be any of the following: - `False` (or anything Falsy): No logging is used. - A `str` or `pathlib.Path`: Logging is used at default options. - An instance of `optimagic.Logger`. There will be multiple subclasses, e.g. `SqliteLogger` which allow us to switch out the logging backend. Each subclass might have different optional arguments. The `log_options` are deprecated. Using dictionaries instead of `Option` objects will be supported during a deprecation cycle. ##### Scaling, error handling and multistart In contrast to logging, scaling, error handling and multistart are deeply baked into optimagic's minimize function. Therefore, it does not make sense to create abstractions for these features that would make them replaceable components that can be switched out for other implementations by advanced users. Most of these features are already perceived as advanced and allow for a lot of configuration. We therefore suggest the following argument types: - `scaling`: `bool | ScalingOptions` - `error_handling`: `bool | ErrorHandlingOptions` - `multistart`: `bool | MultistartOptions` All of the Option objects are simple dataclasses that mirror the current dictionaries. All `_options` arguments are deprecated. ##### `numdiff_options` and similar Dictionaries are still supported but we also offer more autocomplete friendly dataclasses as alternative. (algorithm-interface)= ### The internal algorithm interface and `Algorithm` objects #### Current situation Currently, algorithms are defined as `minimize` functions that are decorated with `om.mark_minimizer`. The `minimize` function returns a dictionary with a few mandatory and several optional keys. Algorithms can provide information to optimagic in two ways: 1. The signature of the minimize function signals whether the algorithm needs derivatives and whether it supports bounds and nonlinear constraints. Moreover, it signals which algorithm specific options are supported. Default values for algorithm specific options are also defined in the signature of the minimize function. 1. `@mark_minimizer` collects the following information via keyword arguments: - Is the algorithm a scalar, least-squares or likelihood optimizer? - The algorithm name. - Does the algorithm require well scaled problems? - Is the algorithm currently installed? - Is the algorithm global or local? - Should the history tracking be disabled (e.g. because the algorithm tracks its own history)? - Does the algorithm parallelize criterion evaluations? A slightly simplified example of the current internal algorithm interface is: ```python @mark_minimizer( name="scipy_neldermead", needs_scaling=False, primary_criterion_entry="value", is_available=IS_SCIPY_AVAILABLE, is_global=False, disable_history=False, ) def scipy_neldermead( criterion, x, lower_bounds, upper_bounds, *, stopping_max_iterations=1_000_000, stopping_max_criterion_evaluations=1_000_000, convergence_absolute_criterion_tolerance=1e-8, convergence_absolute_params_tolerance=1e-8, adaptive=False, ): options = { "maxiter": stopping_max_iterations, "maxfev": stopping_max_criterion_evaluations, # both tolerances seem to have to be fulfilled for Nelder-Mead to converge. # if not both are specified it does not converge in our tests. "xatol": convergence_absolute_params_tolerance, "fatol": convergence_absolute_criterion_tolerance, "adaptive": adaptive, } res = scipy.optimize.minimize( fun=criterion, x0=x, bounds=_get_scipy_bounds(lower_bounds, upper_bounds), method="Nelder-Mead", options=options, ) return process_scipy_result(res) ``` The first two arguments (`criterion` and `x`) are mandatory. The lack of any arguments related to derivatives signifies that `scipy_neldermead` is a gradient free algorithm. The bounds related arguments show that it supports box constraints. The remaining arguments define the supported stopping criteria and algorithm options as well as their default values. The decorator simply attaches information to the function as `_algorithm_info` attribute. This originated as a hack but was never changed afterwards. The `AlgorithmInfo` looks as follows: ```python class AlgoInfo(NamedTuple): primary_criterion_entry: str name: str parallelizes: bool needs_scaling: bool is_available: bool arguments: list # this is read from the signature is_global: bool = False disable_history: bool = False ``` **Things we want to keep** - The internal interface has proven flexible enough for many optimizers we had not wrapped when we designed it. It is easy to add more optional arguments to the decorator without breaking any existing code. - The decorator approach completely hides how we represent algorithms internally. - Since we read a lot of information from function signatures (as opposed to registering options somewhere), there is no duplicated information. If we change the approach to collecting information, we still need to ensure there is no duplication or possibility to provide wrong information to optimagic. **Problems** - Type checkers complain about the `._algorithm_info` hack. - All computations and signature checking are done eagerly for all algorithms at import time. This is one of the reasons why imports are slow. - The first few arguments to the minimize functions follow a naming scheme and any typo in those names would lead to situations that are hard to debug (e.g. if `lower_bound` was miss-typed as `lower_buond` we would assume that the algorithm does not support lower bounds but has a tuning parameter called `lower_buond`). #### Proposal We first show the proposed new algorithm interface and discuss the changes later. ```python @om.mark.minimizer( name="scipy_neldermead", needs_scaling=False, problem_type=om.ProblemType.Scalar, is_available=IS_SCIPY_AVAILABLE, is_global=False, disable_history=False, needs_derivatives=False, needs_parallelism=False, supports_bounds=True, supports_linear_constraints=False, supports_nonlinear_constraints=False, ) @dataclass(frozen=True) class ScipyNelderMead(Algorithm): stopping_max_iterations: int = 1_000_000 stopping_max_criterion_evaluations: int = 1_000_000 convergence_absolute_criterion_tolerance: float = 1e-8 convergence_absolute_params_tolerance: float = 1e-8 adaptive = False def __post_init__(self): # check everything that cannot be handled by the type system assert self.convergence_absolute_criterion_tolerance > 0 assert self.convergence_absolute_params_tolerance > 0 def _solve_internal_problem( self, problem: InternalProblem, x0: NDArray[float] ) -> InternalOptimizeResult: options = { "maxiter": self.stopping_max_iterations, "maxfev": self.stopping_max_criterion_evaluations, "xatol": self.convergence_absolute_params_tolerance, "fatol": self.convergence_absolute_criterion_tolerance, "adaptive": self.adaptive, } res = minimize( fun=problom.scalar.fun, x0=x, bounds=_get_scipy_bounds(problom.bounds), method="Nelder-Mead", options=options, ) return process_scipy_result(res) ``` 1. The new internal algorithms are dataclasses, where all algorithm options are dataclass fields. This enables us to obtain information about the options via the `__dataclass_fields__` attribute without inspecting signatures or imposing naming conventions on non-option arguments. 1. The `_solve_internal_problem` method receives an instance of `InternalProblem` and `x0` (the start values) as arguments. `InternalProblem` collects the criterion function, its derivatives, bounds, etc. This again avoids any potential for typos in argument names. 1. The `mark.minimizer` decorator collects all the information that was previously collected via optional arguments with naming conventions. This information is available while constructing the instance of `InternalProblem`. Thus we can make sure that attributes that were not requested (e.g. derivatives if `needs_derivative` is `False`) raise an `AttributeError` if used. 1. The minimize function returns an `InternalOptimizeResult` instead of a dictionary. The copy constructors (`with_option`, `with_convergence`, and `with_stopping`) are inherited from `optimagic.Algorithm`. This means, that they will have `**kwargs` as signature and thus do not support autocomplete. However, they can check that all specified options are actually in the `__dataclass_fields__` and thus provide feedback before an optimization is run. All breaking changes of the internal algorithm interface are done without deprecation cycle. ```{note} The `_solve_internal_problem` method is private because users should not call it; This also prepares adding a public `minimize` method that internally calls the `minimize` function. ``` To make things more concrete, here are prototypes for components related to the `InternalProblem` and `InternalOptimizeResult`. ```{note} The names of the internal problem are already aligned with the new names for the objective function and its derivatives. ``` ```python from numpy.typing import NDArray from dataclasses import dataclass from typing import Callable, Tuple import optimagic as om @dataclass(frozen=True) class ScalarProblemFunctions: fun: Callable[[NDArray[float]], float] jac: Callable[[NDArray[float]], NDArray[float]] fun_and_jac: Callable[[NDArray[float]], Tuple[float, NDArray[float]]] @dataclass(frozen=True) class LeastSquaresProblemFunctions: fun: Callable[[NDArray[float]], NDArray[float]] jac: Callable[[NDArray[float]], NDArray[float]] fun_and_jac: Callable[[NDArray[float]], Tuple[NDArray[float], NDArray[float]]] @dataclass(frozen=True) class LikelihoodProblemFunctions: fun: Callable[[NDArray[float]], NDArray[float]] jac: Callable[[NDArray[float]], NDArray[float]] fun_and_jac: Callable[[NDArray[float]], Tuple[NDArray[float], NDArray[float]]] @dataclass(frozen=True) class InternalProblem: scalar: ScalarProblemFunctions least_squares: LeastSquaresProblemFunctions likelihood: LikelihoodProblemFunctions bounds: om.Bounds | None linear_constraints: list[om.LinearConstraint] | None nonlinear_constraints: list[om.NonlinearConstraint] | None ``` The `InternalOptimizeResult` formalizes the current dictionary solution: ```python @dataclass(frozen=True) class InternalOptimizeResult: solution_x: NDArray[float] solution_criterion: float n_criterion_evaluations: int | None n_derivative_evaluations: int | None n_iterations: int | None success: bool | None message: str | None ``` #### Alternative to `mark.minimizer` Instead of collecting information about the optimizers via the `mark.minimizer` decorator, we could require the `Algorithm` subclasses to provide that information via class variables. The presence of all required class variables could be enforced via `__init_subclass__`. The two approaches are equivalent in terms of achievable functionality. I see the following advantages and disadvantages: **Advantages of decorator approach** - Easier for beginners as no subtle concepts (such as the difference between instance and class variables) are involved - Very easy way to provide default values for some of the collected variables - Every user of optimagic is familiar with `mark` decorators - Autocomplete while filling out the arguments of the mark decorator - Very clear visual separation of algorithm options and attributes optimagic needs to know about. **Advantages of class variable approach** - More familiar for people with object oriented background - Possibly better ways to enforce the presence of the class variables via static analysis I am personally leaning towards the decorator approach but any feedback on this topic is welcome. ## Numerical differentiation ### Current situation The following proposal applies to the functions `first_derivative` and `second_derivative`. Both functions have an interface that has grown over time and both return a relatively complex result dictionary. There are several arguments that govern which entries are stored in the result dictionary. The functions `first_derivative` and `second_derivative` allow params to be arbitrary pytrees. They work for scalar and vector valued functions and a `key` argument makes sure that they work for `criterion` functions that return a dict containing `"value"`, `"contributions"`, and `"root_contributions"`. In contrast to optimization, all pytree handling (for params and function outputs) is mixed with the calculation of the numerical derivatives. This can produce more informative error messages and save some memory. However it increases complexity extremely because we can make very few assumptions on types. There are many if conditions to deal with this situation. The interface is further complicated by supporting Richardson Extrapolation. This feature was inspired by [numdifftools](https://numdifftools.readthedocs.io/en/latest/) but has not produced convincing results in benchmarks. **Things we want to keep** - `params` and function values can be pytrees - support for optimagic `criterion` functions (now functions that return `FunctionValue`) - Many optional arguments to influence the details of the numerical differentiation - Rich output format that helps to get insights on the precision of the numerical differentiation - Ability to optionally pass in a function evaluation at `params` or return a function evaluation at `params` **Problems** - We can make no assumptions on types inside the function because pytree handling is mixed with calculations - Support for Richardson extrapolation complicates the interface and implementation but has not been convincing in benchmarks - Pytree handling is acatually incomplete (`base_steps`, `min_steps` and `step_ratio` are assumed to be flat numpy arrays) - Many users expect the output of a function for numerical differentiation to be just the gradient, jacobian or hessian, not a more complex result object. ### Proposal #### Separation of calculations and pytree handling As in numerical optimization, we should implement the core functionality for first and second derivative for functions that map from 1-Dimensional numpy arrays to 1-Dimensional numpy arrays. All pytree handling or other handling of function outputs (e.g. functions that return a `FunctionValue`) should be done outside of the core functions. #### Deprecate Richardson Extrapolation (and prepare alternatives) The goal of implementing Richardson Extrapolation was to get more precise estimates of numerical derivatives when it is hard to find an optimal step size. Example use-cases we had in mind were: - Optimization of a function that is piecewise flat, e.g. the likelihood function of a naively implemented multinomial probit - Optimization or standard error estimation of slightly noisy functions, e.g. functions of an MSM estimation problem - Standard error estimation of wiggly functions where the slope and curvature at the minimum does not yield reasonable standard errors and confidence intervals Unfortunately, the computational cost of Richardson extrapolation is too high for any application during optimization. Moreover, our practical experience with Richardson Extrapolation was not positive and it seems that Richardson extrapolation is not designed for our use-cases. It is designed as a sequence acceleration method that reduces roundoff error while shrinking a step size to zero, whereas in our application it might often be better to take a larger step size (for example, the success of derivative free trust-region optimizers suggest less local slope and curvature information is more useful than actual derivatives for optimization; similarly, numerical derivatives with larger step sizes could be seen as an estimate of a [quasi jacobian](https://arxiv.org/abs/1907.13093) and inference based on it might have good statistical properties). We therefore propose to remove Richardson extrapolation and open an Issue to work on alternatives. Examples for alternatives could be: - [Moré and Wild (2010)](https://www.mcs.anl.gov/papers/P1785.pdf) propose an approach to calculate optimal step sizes for finite difference differentiation of noisy functions - We could think about aggregating derivative estimates at multiple step sizes in a way that produces worst case standard errors and confidence intervals - ... ```{note} Richardson extrapolation was only completed for first derivatives, even though it is already prepared in the interface for second derivatives. ``` #### Better `NumdiffResult` object The result dictionary will be replaced by a `NumdiffResult` object. All arguments that govern which results are stored will be removed. If some of the formerly optional results require extra computation that we wanted to avoid by making them optional, they can be properties or methods of the result object. #### Jax inspired high-level interfaces Since our `first_derivative` and `second_derivative` functions need to fulfill very specific requirements for use during optimization, they need to return a complex result object. However, this can be annoying in simple situations where users just want a gradient, jacobian or hessian. To cover these simple situations and provide a high level interface to our numdiff functions, we can provide a set of jax inspired decorators: - `@grad` - `@value_and_grad` - `@jac` (no distinction between `@jacrev` and `jacfwd` necessary) - `@value_and_jac` - `@hessian` - `@value_and_hessian` All of these will be very simple wrappers around `first_derivative` and `second_derivative` with very low implementation and maintenance costs. ## Benchmarking ### `get_benchmark_problems` #### Current situation As other functions in optimagic, `get_benchmark_problems` follows a design where behavior can be switched on by a bool and configured by an options dictionary. The following arguments are related to this: - `additive_noise` and `additive_noise_options` - `multiplicative_noise` and `multiplicative_noise_options` - `scaling` and `scaling_options` All of them have the purpose of adding some difficult characteristics to an existing benchmark set, so we can analyze how well an optimizer can deal with this situation. The name of the benchmark set is passed in as a string. The return value of `get_benchmark_problems` is a nested dictionary. The keys in the outer dictionary are the names of benchmark problems. The inner dictionaries represent benchmark problems. **Things we want to keep** - Benchmark problems are collected in a dict, not in a fixed-field data structure. This makes it easy to merge problems from multiple benchmark sets or filter benchmark sets. A fixed field data structure would not work here. **Problems** - As discussed before, having separate arguments for switching-on behavior and configuring it can be dangerous - Each single benchmark problem should not be represented as a dictionary - Adding noise or scaling problems should be made more flexible and generic #### Proposal ##### Add noise to benchmark problems The four arguments `additive_noise`, `multiplicative_noise`, `additive_noise_options`, and `multiplicative_noise_options` are combined in one `noise` argument. This `noise` argument can be `bool | BenchmarkNoise`. If `False`, no noise is added. If `True`, standard normal noise is added. We implement several subclasses of `BenchmarkNoise` to cover the current use cases. As syntactic sugar, we can make `BenchmarkNoise` instances addable (by implementing an `__add__` method) so multiple sources of noise can be combined. A rough prototype for `BenchmarkNoise` looks as follows: ```python FvalType = TypeVar("FvalType", bound=float | NDArray[float]) class BenchmarkNoise(ABC): @abstractmethod def draw_noise( self, fval: FvalType, params: NDArray, size: int, rng: np.random.Generator ) -> FvalType: pass def __add__(self, other: BenchmarkNoise): pass ``` Passing `fval` and `params` to `draw_noise` enables use to implement multiplicative noise (i.e. noise where the standard deviation scales with the function value) and stochastic or deterministic wiggle (e.g. a sine curve that depends on params). Therefore, this proposal does not just cover everything that is currently implemented but also adds new functionality we wanted to implement. ##### Add scaling issues to benchmark problems The `scaling_options` argument is deprecated. The `scaling` argument can be `bool | BenchmarkScaler`. We implement `LinspaceBenchmarkScaler` to cover everything that is implemented right now but more types of scaling can be implemented in the future. A rough prototype of `BenchmarkScaler` looks as follows: ```python class BenchmarkScaler(ABC): @abstractmethod def scale(self, params: NDArray) -> NDArray: pass @abstractmethod def unscale(self, params: NDArray) -> NDArray: pass ``` ##### Representing benchmark problems Instead of the fixed-field dictionary we will have a dataclass with corresponding fields. This would roughly look as follows: ```python @dataclass class BenchmarkProblem: fun: Callable[[NDArray], FunctionValue] start_x: NDArray solution_x: NDArray | None start_fun: float solution_fun: float ``` ### `run_benchmark` #### Current situation `run_benchmark` takes `benchmark_problems` (covered in the previous section), `optimize_options` and a few other arguments and returns a nested dictionary representing benchmark results. `optimize_options` can be a list of algorithm names, a dict with algorithm names as values or a nested dict of keyword arguments for `minimize`. **Things we want to keep** - Benchmark results are collected in a dict, not in a fixed-field data structure. This makes it easy to merge results from multiple benchmark sets or filter benchmark results. A fixed field data structure would not work here. **Problems** - `optimize_options` are super flexible but error prone and hard to write as there is no autocomplete support - Each single benchmark result should not be represented as a dictionary #### Proposal We restrict the typo of `optimize_options` to `dict[str, Type[Algorithm] | Algorithm | OptimizeOptions]`. Here, `OptimizeOptions` will be a simple dataclass that we need for `estimate_ml` and `estimate_msm` anyways. Passing just lists of algorithm names is deprecated. Passing dicts as optimize options is also deprecated. Most use-cases will be covered by passing dictionaries of configured Algorithms as optimize options. Actually using the full power of passing `OptimizeOptions` will be rarely needed. The return type of `run_benchmark` will be `dict[tuple[str], BenchmarkResult]` `BenchmarkResult` is a dataclass with fields that mirror the keys of the current dictionary. It will roughly look as follows: ```python @dataclass class BenchmarkResult: params_history: list[NDArray] fun_history: list[float] time_history: list[float] batches_history: list[int] solution: OptimizeResult ``` ## Estimation The changes to the estimation functions `estimate_ml` and `estimate_msm` will be minimal: - `lower_bounds` and `upper_bounds` are replaced by `bounds` (as in optimization) - `numdiff_options` and `optimize_options` become dataclasses - `logging` and `log_options` get aligned with our proposal for optimization In the long run we plan a general overhaul of `MSM` estimation that provides better access to currently internal objects such as the MSM objective function. ## Type checkers and their configuration We choose mypy as static type checker and run it as part of our continuous integration. Once this enhancement proposal is fully implemented, we want to use the following settings: ``` check_untyped_defs = true disallow_any_generics = true disallow_untyped_defs = true disallow_incomplete_defs = true no_implicit_optional = true warn_redundant_casts = true warn_unused_ignores = true ``` In addition to CI, we could also run type-checks as part of the pre-commit hooks. An example where this is done can be found [here](https://github.com/google/jax/blob/de0fd722f0c4c0c238884f0e64e4ef8da72e4c1d/.pre-commit-config.yaml#L33). ## Runtime type checking Since most of our users do not use static type checkers we will still need to check the type of most user inputs so we can give them early feedback when problems arise. Thus we cannot remove our current error handling just because many of these errors could now be caught by static analysis. We can investigate using `jaxtyping`'s pytest hooks to enable runtime typecheckers like beartype during testing but it is not a priority for now. ## Changes in documentation All type information in docstrings will be removed. Whenever there are now multiple ways of doing things, we show the ones that support autocomplete and static analysis most prominently. We can achieve this via tabs, similar to how [pytask](https://pytask-dev.readthedocs.io/en/stable/tutorials/defining_dependencies_products.html#products) does it. The general structure of the documentation is not affected by this enhancement proposal. ## Summary of breaking changes - The internal algorithm interface changes completely without deprecations - The support for Richardson Extrapolation in `first_derivative` is dropped without deprecation; The corresponding arguments `n_steps` and `step_ratio` are removed. - The return type of `first_derivative` and `second_derivative` changes from dict to `NumdiffResult` without deprecations. The arguments `return_func_value` and `return_info` are dropped. - The representation of benchmark problems and benchmark results changes without deprecations ## Summary of deprecations The following deprecations become active in version `0.5.0`. The functionality will be removed in version `0.6.0` which should be scheduled for approximately half a year after the realease of `0.5.0`. - Returning a `dict` in the objective function io deprecated. Return `FunctionValue` instead. In addition, likelihood and least-squares problems need to be decorated with `om.mark.likelihood` and `om.mark_least_squares`. - The arguments `lower_bounds`, `upper_bounds`, `soft_lower_bounds` and `soft_upper_bounds` are deprecated. Use `bounds` instead. `bounds` can be `optimagic.Bounds` or `scipy.optimize.Bounds` objects. - Specifying constraints with dictionaries is deprecated. Use the corresponding subclass of `om.constraints.Constraint` instead. In addition, all selection methods except for `selector` are deprecated. - The `covariance` constraint is renamed to `FlatCovConstraint` and the `sdcorr` constraint is renamed to `FlatSdcorrConstraint` to prepare the introduction of more natural (non-flattened) covariance and sdcorr constraints. - The `log_options` argument of `maximize` and `minimize` is deprecated and gets subsumed in the `logging` argument. - The `scaling_options` argument of `maximize` and `minimize` is deprecated and gets subsumed in the `scaling` argument. - The `error_penalty` argument of `maximize` and `minimize` is deprecated and gets subsumed in the `error_handling` argument. - The `multistart_options` argument of `maximize` and `minimize` is deprecated and gets subsumed in the `multistart` argument. - The arguments `additive_noise`, `additive_noise_options`, `multiplicative_noise`, and `multiplicative_noise_options` in `get_benchmark_problems` are deprecated and combined into `noise`. - The `scaling_options` argument in `get_benchmark_problems` is deprecated and subsumed in the `scaling` argument. - Passing just a list of algorithm strings as `optimize_options` in `run_benchmark` is deprecated. (eepalignment)= # EP-03: Alignment with SciPy ```{eval-rst} +------------+------------------------------------------------------------------+ | Author | `Janos Gabler `_ | +------------+------------------------------------------------------------------+ | Status | Accepted | +------------+------------------------------------------------------------------+ | Type | Standards Track | +------------+------------------------------------------------------------------+ | Created | 2024-07-09 | +------------+------------------------------------------------------------------+ | Resolution | | +------------+------------------------------------------------------------------+ ``` ## Abstract This enhancement proposal explains how we will better align optimagic with `scipy.minimize`. Scipy is the most widely used optimizer library in Python and most of our new users are switching over from SciPy. The goal is therefore simple: Make it as easy as possible for SciPy users to use optimagic. In most cases this means that the only thing that has to be changed is the import statement for the `minimize` function: ```python # from scipy.optimize import minimize from optimagic import minimize ``` ## Design goals - If we can make code written for SciPy run with optimagic, we should do so - If we cannot make it run, the user should get a helpful error message that explains how the code needs to be adjusted. ## Aligning names | **Old Name** | **Proposed Name** | **Source** | | ------------------------------------------ | ------------------------- | ---------- | | `criterion` | `fun` | scipy | | `criterion_kwargs` | `fun_kwargs` | | | `params` | `x0` | | | `derivative` | `jac` | scipy | | `derivative_kwargs` | `jac_kwargs` | | | `criterion_and_derivative` | `fun_and_jac` | | | `criterion_and_derivative_kwargs` | `fun_and_jac_kwargs` | | | `stopping_max_criterion_evaluations` | `stopping_maxfun` | scipy | | `stopping_max_iterations` | `stopping_maxiter` | scipy | | `convergence_absolute_criterion_tolerance` | `convergence_ftol_abs` | NlOpt | | `convergence_relative_criterion_tolerance` | `convergence_ftol_rel` | NlOpt | | `convergence_absolute_params_tolerance` | `convergence_xtol_abs` | NlOpt | | `convergence_relative_params_tolerance` | `convergence_xtol_rel` | NlOpt | | `convergence_absolute_gradient_tolerance` | `convergence_gtol_abs` | NlOpt | | `convergence_relative_gradient_tolerance` | `convergence_gtol_rel` | NlOpt | | `convergence_scaled_gradient_tolerance` | `convergence_gtol_scaled` | | While it seems that many names are taken from NlOpt and not from SciPy, this is a bit misleading. SciPy does use the words `xtol`, `ftol` and `gtol` just like NlOpt, but it does not completely harmonize them between algorithms. We therefore chose NlOpt's version which is understandable for everyone who knows SciPy but more readable than SciPy's. ## Names we do not want to align - We do not want to rename `algorithm` to `method` because our algorithm names are different from SciPy, so people who switch over from SciPy need to adjust their code anyways. - We do not want to rename `algo_options` to `options` for the same reason. Instead we can provide aliases for those. ## Additional aliases To make it even easier for SciPy users to switch to optimagic, we can provide additional aliases in `minimize` and `maximize` that let them used their SciPy code without changes or help to adjust it by showing good error messages. The following arguments are relevant: - `method`: In SciPy this is used instead of `algorithm` to select the optimization algorithm. We opted against simply renaming `algorithm` to `method` because our naming scheme of algorithms is (and has to be) different from SciPy. By using `method` instead of `algorithm`, users could select SciPy algorithms by their SciPy name. If `method` and `algorithm` are both provided, they would get an error. - `tol`: We do not want to support one `tol` argument for all kinds of different convergence criteria but could raise an error for people who use it and point them to the relevant parts of our documentation. - `args`: we can support `args` as an alternative to `fun_kwargs` - `options`: This is the SciPy counterpart to our `algo_options`. We do not want to support this as our option names are different but we can provide a good error message with pointers to our documentation if someone uses it. - `hess` and `hessp`: Currently we don't support closed form hessians. If we support them they will be called `hess`. In the meantime, this can raise a `NotImplementedError`. - `callback`: Currently we do not support `callback`s. If we support them they will be called `callback` and be as compatible with SciPy as possible. In the meantime we can raise a `NotImplementedError`. - If a user sets `jac=True` we raise and error and explain how to use `fun_and_jac` instead. ## Letting algorithms pick their default values Currently we try to align default values for convergence criteria and other algorithm options across algorithms and even across optimizer packages. This means that sometimes algorithms that are used via optimagic produce different results than the same algorithm used via SciPy or other packages. Moreover, it is possible that we deviate from algorithm options that the original authors carefully picked because they maximize performance on a relevant benchmark set. I therefore propose that in the future we do not try to align algorithm options across algorithms and packages. ## Implementation All renamings are done with a careful deprecation cycle. The deprecations become active in version `0.5.0`. Old names will be removed in version `0.6.0` which should be scheduled for approximately half a year after the release of `0.5.0`. (how-to-contribute)= # How to contribute ## 1. Intro We welcome and greatly appreciate contributions of all forms and sizes! Whether it's updating the documentation, adding small extensions, or implementing new features, every effort is valued. For substantial changes, please contact us in advance. This allows us to discuss your ideas and guide the development process from the beginning. You can start a conversation by posting an issue on GitHub or by emailing [janosg](https://github.com/janosg). To get familiar with the codebase, we recommend checking out our [issue tracker](https://github.com/optimagic-dev/optimagic/issues) for some immediate and clearly defined tasks. ## 2. Before you start Once you've decided to contribute, please review the {ref}`style_guide` (see the next page) to ensure your work aligns with the project's coding standards. We manage new features through Pull Requests (PRs). Contributors work on their local copy of optimagic, modifying and extending the codebase there, before opening a PR to propose merging their changes into the main branch. Regular contributors gain push access to unprotected branches, which simplifies the contribution process (see Notes below). ## 3. Step-by-step guide 1. Fork the [optimagic repository](https://github.com/optimagic-dev/optimagic/). This action creates a copy of the repository with write access for you. ```{note} For regular contributors: **Clone** the [repository](https://github.com/optimagic-dev/optimagic/) to your local machine and create a new branch for implementing your changes. You can push your branch directly to the remote optimagic repository and open a PR from there. ``` 1. Clone your forked repository to your disk. This is where you'll make all your changes. 1. Open your terminal and execute the following commands from the root directory of your local optimagic repository: ```console $ prek install ``` This activates pre-commit hooks for linting and style formatting. ```{note} `prek` is not managed by pixi and must be installed globally. You can find installation instructions at [github.com/j178/prek](https://github.com/j178/prek). ``` You can then run the test suite with: ```console $ pixi run tests ``` which installs the development dependencies and runs pytest. To run the type checker, use: ```console $ pixi run mypy ``` To see all available pixi tasks, run: ```console $ pixi task list ``` 1. Implement your fix or feature. Use git to add, commit, and push your changes to the remote repository. For more on git and how to stage and commit your work, refer to these [online materials](https://effective-programming-practices.vercel.app/git/staging/objectives_materials.html). 1. Contributions are validated in two main ways. We run a comprehensive test suite to ensure compatibility with the existing codebase and employ [pre-commit hooks](https://effective-programming-practices.vercel.app/git/pre_commits/objectives_materials.html) to maintain quality and adherence to our style guidelines. Opening a PR (see below) triggers optimagic's [Continuous Integration (CI)](https://docs.github.com/en/actions/automating-builds-and-tests/about-continuous-integration) workflow, which runs the full test suite, pre-commit hooks, and other checks on a remote server. You can also run the test suite locally for [debugging](https://effective-programming-practices.vercel.app/debugging/pdbp/objectives_materials.html). With prek installed, linters run before each commit. Commits are rejected if any checks fail. Note that some linters may automatically fix errors by modifying the code in-place. Remember to re-stage the files after such modifications. ```{tip} Skip the next paragraph if you haven't worked on the documentation. ``` 1. Assuming you have updated the documentation, verify that it builds correctly. Run: ```console $ pixi run build-docs ``` This command builds the HTML documentation, saving all files in the `docs/build/html` directory. You can view the documentation with your preferred web browser by opening `docs/build/html/index.html` or any other file. Similar to the online documentation, you can navigate to different pages simply by clicking on the links. 1. Once all tests and hooks pass locally, push your changes to your forked repository and create a pull request through GitHub: Go to the Github repository of your fork. A banner on your fork's GitHub repository will prompt you to open a PR. ```{note} Regular contributors with push access can directly push their local branch to the remote optimagic repository and initiate a PR from there. ``` Follow the steps outlined in the optimagic [PR template](https://github.com/optimagic-dev/optimagic/blob/main/.github/PULL_REQUEST_TEMPLATE/pull_request_template.md) to describe your contribution, the problem it addresses, and your proposed solution. Opening a PR initiates a complete CI run, including the `pytest` suite, linters, code coverage checks, doctests, and building the HTML documentation. Monitor the CI workflow status on your PR page and make necessary modifications to your code based on the results, iterating until all tests pass. 1. Request a review from one of the main contributors once all CI tests pass. Address any feedback or suggestions by making the necessary changes and committing them. 1. After your PR is approved, one of the main contributors will merge it into optimagic's main branch. (style_guide)= # Styleguide Your contribution should fulfill the criteria provided below. ## Styleguide for the codebase - Functions have no side effect. : If you modify a mutable argument, make a copy at the beginning of the function. - Use good names for functions and variables : *"You should name a variable using the same care with which you name a first-born child."*, Robert C. Martin, Clean Code: A Handbook of Agile Software Craftsmanship. A bit more concretely, this means: The length of a variable name should be proportional to its scope. In a list comprehension or short loop, i might be an acceptable name for the running variable, but variables that are used at many different places should have descriptive names. The name of variables should reflect the content or meaning of the variable and not only the type. Names like `dict_list` would not have been a good name for the `constraints`. Function names should contain a verb. Moreover, the length of a function name is typically inversely proportional to its scope. The public functions like `maximize` and `minimize` can have very short names. At a lower level of abstraction you typically need more words to describe what a function does. - User facing functions should be generous regarding their input type. Example: the `algorithm` argument can be a string, `Algorithm` class or `Algorithm` instance. The `algo_options` can be an `AlgorithmOptions` object or a dictionary of keyword arguments. - User facing functions should be strict about their output types. A strict output type does not just mean that the output type is known (and not a generous Union), but that it is a proper type that enables static analysis for available attributes. Example: whenever possible, public functions should not return dicts but proper result types (e.g. `OptimizeResult`, `NumdiffResult`, ...) - Internal functions should be strict about input and output types; Typically, a public function will check all arguments, convert them to a proper type and then call an internal function. Example: `minimize` will convert any valid value for `algorithm` into an `Algorithm` instance and then call an internal function with that type. - Fixed field types should only be used if all fields are known. An example where this is not the case are collections of benchmark problems, where the set of fields depends on the selected benchmark sets and other things. In such situations, dictionaries that map strings to BenchmarkProblem objects are a good idea. - Think about autocomplete! If you want to accept a string as an argument (e.g. an algorithm name) also accept input types that are more amenable to static analysis and offer better autocomplete. - Whenever possible, use immutable types. Whenever things need to be changeable, consider using an immutable type with copy constructors for modified instances. Example: instances of `Algorithm` are immutable but using `Algorithm.with_option` users can create modified copies. - The main entry point to optimagic are functions, objects are mostly used for configuration and return types. This takes the best of both worlds: we get the safety and static analysis that (in Python) can only be achieved using objects but the beginner friendliness and freedom provided by functions. Example: Having a `minimize` function, it is very easy to add the possibility of running minimizations with multiple algorithms in parallel and returning the best value. Having a `.solve` method on an algorithm object would require a whole new interface for this. - Deep modules. : This is a term coined by [John Ousterhout](https://www.youtube.com/watch?v=bmSAYlu0NcY). A deep module is a module that has just one public function. This function calls the private functions (i.e. functions that start with an underscore) defined further down in the module and reads almost like a table of contents to the whole module. - Never import a private function in another module : By following this strictly, you can be sure that you can rename or refactor private functions without looking at other modules. Of course it is also not a solution to copy paste the function! If you would like to import a function that starts with an underscore, rename it. - All functions have a [Google style](https://tinyurl.com/mxams9k) docstring : The docstring describes all arguments and outputs. For arrays, please document how many dimensions and what shape they have. Look around in the code to find examples if you are in doubt. Example: ```python def ordered_logit(formula, data): """Estimate an ordered probit model with maximum likelihood. Args: formula (str): A patsy formula. data (str): A pandas DataFrame. Returns: res: optimization result. """ pass ``` In particular each docstring should start with a one liner that describes very concisely what the function does. The one liner should be in imperative mode, i.e. not "This function does" ..." , but "Do ..." and end with a period. - Unit tests : If you write a small helper whose interface might change during refactoring, it is sufficient if the function that calls it is tested. But all functions that are exposed to the user must have unit tests. - Enable pre-commit hooks by executing `prek install` in a terminal in the root of the optimagic repository. This makes sure that your formatting is consistent with what we expect. - Use `pathlib` for all file paths operations. : You can find the pathlib documentation [here](https://docs.python.org/3/library/pathlib.html) - Object serialization. : Pickling and unpickling of DataFrames should be done with `pd.read_pickle` and `pd.to_pickle`. - Don't use global variables unless absolutely necessary : Exceptions are global variables from a config file that replace magic numbers. Never use mutable global variables! ## Styleguide for the documentation - General. : The documentation is rendered with [Sphinx](https://www.sphinx-doc.org/en/master/) and written in **Markedly Structured Text.** How-to guides are usually Jupyter notebooks. - The documentation follows the [diataxis](https://diataxis.fr) framework. (list_of_videos)= # Videos Check out our tutorials, talks and screencasts about optimagic. ## Talks and tutorials ### EuroSciPy 2023 (Talk) ```{raw} html ``` ### EuroSciPy 2023 (Tutorial) ```{raw} html ``` ### SciPy 2022 (Tutorial) ```{raw} html ``` ## Screencasts The screencasts are part of the course _Effective Programming Practices for Economists_, taught at the University of Bonn by [Hans-Martin von Gaudecker](https://www.wiwi.uni-bonn.de/gaudecker/), and previously also [Janoś Gabler](https://github.com/janosg). You can find all screencasts of the course on the [course webite](https://effective-programming-practices.vercel.app/landing-page.html). Here, we show the screencasts about numerical optimization and optimagic. ### Introduction to numerical optimization ```{raw} html ``` ### Using optimagic’s minimize and maximize ```{raw} html ``` ### Visualizing optimizer histories ```{raw} html ``` ### Choosing optimization algorithms ```{raw} html ``` (list_of_algorithms)= # Optimizers Check out {ref}`how-to-select-algorithms` to see how to select an algorithm and specify `algo_options` when using `maximize` or `minimize`. The default algorithm options are discussed in {ref}`algo_options` and their type hints are documented in {ref}`typing`. ## Optimizers from SciPy (scipy-algorithms)= optimagic supports most [SciPy](https://scipy.org/) algorithms and SciPy is automatically installed when you install optimagic. ```{eval-rst} .. dropdown:: scipy_lbfgsb **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.scipy_lbfgsb(stopping_maxiter=1_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="scipy_lbfgsb", algo_options={"stopping_maxiter": 1_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.scipy_optimizers.ScipyLBFGSB ``` ```{eval-rst} .. dropdown:: scipy_slsqp .. code-block:: "scipy_slsqp" Minimize a scalar function of one or more variables using the SLSQP algorithm. SLSQP stands for Sequential Least Squares Programming. SLSQP is a line search algorithm. It is well suited for continuously differentiable scalar optimization problems with up to several hundred parameters. The optimizer is taken from scipy which wraps the SLSQP optimization subroutine originally implemented by :cite:`Kraft1988`. .. note:: SLSQP's general nonlinear constraints are not supported yet by optimagic. - **convergence.ftol_abs** (float): Precision goal for the value of f in the stopping criterion. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **display** (bool): Set to True to print convergence messages. Default is False. Scipy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_neldermead .. code-block:: "scipy_neldermead" Minimize a scalar function using the Nelder-Mead algorithm. The Nelder-Mead algorithm is a direct search method (based on function comparison) and is often applied to nonlinear optimization problems for which derivatives are not known. Unlike most modern optimization methods, the Nelder–Mead heuristic can converge to a non-stationary point, unless the problem satisfies stronger conditions than are necessary for modern methods. Nelder-Mead is never the best algorithm to solve a problem but rarely the worst. Its popularity is likely due to historic reasons and much larger than its properties warrant. The argument `initial_simplex` is not supported by optimagic as it is not compatible with optimagic's handling of constraints. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **convergence.xtol_abs** (float): Absolute difference in parameters between iterations that is tolerated to declare convergence. As no relative tolerances can be passed to Nelder-Mead, optimagic sets a non zero default for this. - **convergence.ftol_abs** (float): Absolute difference in the criterion value between iterations that is tolerated to declare convergence. As no relative tolerances can be passed to Nelder-Mead, optimagic sets a non zero default for this. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. - **adaptive** (bool): Adapt algorithm parameters to dimensionality of problem. Useful for high-dimensional minimization (:cite:`Gao2012`, p. 259-277). scipy's default is False. ``` ```{eval-rst} .. dropdown:: scipy_powell .. code-block:: "scipy_powell" Minimize a scalar function using the modified Powell method. .. warning:: In our benchmark using a quadratic objective function, the Powell algorithm did not find the optimum very precisely (less than 4 decimal places). If you require high precision, you should refine an optimum found with Powell with another local optimizer. The criterion function need not be differentiable. Powell's method is a conjugate direction method, minimizing the function by a bi-directional search in each parameter's dimension. The argument ``direc``, which is the initial set of direction vectors and which is part of the scipy interface is not supported by optimagic because it is incompatible with how optimagic handles constraints. - **convergence.xtol_rel (float)**: Stop when the relative movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. More formally, this is expressed as .. math:: \frac{(f^k - f^{k+1})}{\\max{{\{|f^k|, |f^{k+1}|, 1\}}}} \leq \text{relative_criterion_tolerance} - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count thisas convergence. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_bfgs .. code-block:: "scipy_bfgs" Minimize a scalar function of one or more variables using the BFGS algorithm. BFGS stands for Broyden-Fletcher-Goldfarb-Shanno algorithm. It is a quasi-Newton method that can be used for solving unconstrained nonlinear optimization problems. BFGS is not guaranteed to converge unless the function has a quadratic Taylor expansion near an optimum. However, BFGS can have acceptable performance even for non-smooth optimization instances. - **convergence.gtol_abs** (float): Stop if all elements of the gradient are smaller than this. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **norm** (float): Order of the vector norm that is used to calculate the gradient's "score" that is compared to the gradient tolerance to determine convergence. Default is infinite which means that the largest entry of the gradient vector is compared to the gradient tolerance. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. - **convergence_xtol_rel** (float): Relative tolerance for `x`. Terminate successfully if step size is less than `xk * xrtol` where `xk` is the current parameter vector. Default is 1e-5. SciPy name: **xrtol**. - **armijo_condition** (float): Parameter for Armijo condition rule. Default is 1e-4. Ensures .. math:: f(x_k+\alpha p_k) \le f(x_k) \;+\mathrm{armijo\_condition}\,\cdot\,\alpha\,\nabla f(x_k)^\top p_k, so each step yields at least a fraction **armijo_condition** of the predicted decrease. Smaller ⇒ more aggressive steps, larger ⇒ more conservative ones. SciPy name: **c1**. - **curvature_condition** (float): Parameter for curvature condition rule. Default is 0.9. Ensures .. math:: \nabla f(x_k+\alpha p_k)^\top p_k \ge \mathrm{curvature\_condition}\,\cdot\,\nabla f(x_k)^\top p_k, so the new slope isn’t too negative. Smaller ⇒ stricter curvature reduction (smaller steps), larger ⇒ looser (bigger steps). SciPy name: **c2**. ``` ```{eval-rst} .. dropdown:: scipy_conjugate_gradient .. code-block:: "scipy_conjugate_gradient" Minimize a function using a nonlinear conjugate gradient algorithm. The conjugate gradient method finds functions' local optima using just the gradient. This conjugate gradient algorithm is based on that of Polak and Ribiere, detailed in :cite:`Nocedal2006`, pp. 120-122. Conjugate gradient methods tend to work better when: - the criterion has a unique global minimizing point, and no local minima or other stationary points. - the criterion is, at least locally, reasonably well approximated by a quadratic function. - the criterion is continuous and has a continuous gradient. - the gradient is not too large, e.g., has a norm less than 1000. - The initial guess is reasonably close to the criterion's global minimizer. - **convergence.gtol_abs** (float): Stop if all elements of the gradient are smaller than this. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **norm** (float): Order of the vector norm that is used to calculate the gradient's "score" that is compared to the gradient tolerance to determine convergence. Default is infinite which means that the largest entry of the gradient vector is compared to the gradient tolerance. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_newton_cg .. code-block:: "scipy_newton_cg" Minimize a scalar function using Newton's conjugate gradient algorithm. .. warning:: In our benchmark using a quadratic objective function, the truncated newton algorithm did not find the optimum very precisely (less than 4 decimal places). If you require high precision, you should refine an optimum found with Powell with another local optimizer. Newton's conjugate gradient algorithm uses an approximation of the Hessian to find the minimum of a function. It is practical for small and large problems (see :cite:`Nocedal2006`, p. 140). Newton-CG methods are also called truncated Newton methods. This function differs scipy_truncated_newton because - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy and scipy while ``scipy_truncated_newton``'s algorithm calls a C function. - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization while ``scipy_truncated_newton``'s algorithm supports bounds. Conjugate gradient methods tend to work better when: - the criterion has a unique global minimizing point, and no local minima or other stationary points. - the criterion is, at least locally, reasonably well approximated by a quadratic function. - the criterion is continuous and has a continuous gradient. - the gradient is not too large, e.g., has a norm less than 1000. - The initial guess is reasonably close to the criterion's global minimizer. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. Newton CG uses the average relative change in the parameters for determining the convergence. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_cobyla .. code-block:: "scipy_cobyla" Minimize a scalar function of one or more variables using the COBYLA algorithm. COBYLA stands for Constrained Optimization By Linear Approximation. It is derivative-free and supports nonlinear inequality and equality constraints. .. note:: Cobyla's general nonlinear constraints is not supported yet by optimagic. Scipy's implementation wraps the FORTRAN implementation of the algorithm. For more information on COBYLA see :cite:`Powell1994`, :cite:`Powell1998` and :cite:`Powell2007`. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. In case of COBYLA this is a lower bound on the size of the trust region and can be seen as the required accuracy in the variables but this accuracy is not guaranteed. - **trustregion.initial_radius** (float): Initial value of the trust region radius. Since a linear approximation is likely only good near the current simplex, the linear program is given the further requirement that the solution, which will become the next evaluation point must be within a radius RHO_j from x_j. RHO_j only decreases, never increases. The initial RHO_j is the `trustregion.initial_radius`. In this way COBYLA's iterations behave like a trust region algorithm. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_truncated_newton .. code-block:: "scipy_truncated_newton" Minimize a scalar function using truncated Newton algorithm. This function differs from scipy_newton_cg because - ``scipy_newton_cg``'s algorithm is written purely in Python using NumPy and scipy while ``scipy_truncated_newton``'s algorithm calls a C function. - ``scipy_newton_cg``'s algorithm is only for unconstrained minimization while ``scipy_truncated_newton``'s algorithm supports bounds. Conjugate gradient methods tend to work better when: - the criterion has a unique global minimizing point, and no local minima or other stationary points. - the criterion is, at least locally, reasonably well approximated by a quadratic function. - the criterion is continuous and has a continuous gradient. - the gradient is not too large, e.g., has a norm less than 1000. - The initial guess is reasonably close to the criterion's global minimizer. optimagic does not support the ``scale`` nor ``offset`` argument as they are not compatible with the way optimagic handles constraints. It also does not support ``messg_num`` which is an additional way to control the verbosity of the optimizer. - **func_min_estimate** (float): Minimum function value estimate. Defaults to 0. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **convergence.xtol_abs** (float): Absolute difference in parameters between iterations after scaling that is tolerated to declare convergence. - **convergence.ftol_abs** (float): Absolute difference in the criterion value between iterations after scaling that is tolerated to declare convergence. - **convergence.gtol_abs** (float): Stop if the value of the projected gradient (after applying x scaling factors) is smaller than this. If convergence.gtol_abs < 0.0, convergence.gtol_abs is set to 1e-2 * sqrt(accuracy). - **max_hess_evaluations_per_iteration** (int): Maximum number of hessian*vector evaluations per main iteration. If ``max_hess_evaluations == 0``, the direction chosen is ``- gradient``. If ``max_hess_evaluations < 0``, ``max_hess_evaluations`` is set to ``max(1,min(50,n/2))`` where n is the length of the parameter vector. This is also the default. - **max_step_for_line_search** (float): Maximum step for the line search. It may be increased during the optimization. If too small, it will be set to 10.0. By default we use scipy's default. - **line_search_severity** (float): Severity of the line search. If < 0 or > 1, set to 0.25. optimagic defaults to scipy's default. - **finitie_difference_precision** (float): Relative precision for finite difference calculations. If <= machine_precision, set to sqrt(machine_precision). optimagic defaults to scipy's default. - **criterion_rescale_factor** (float): Scaling factor (in log10) used to trigger criterion rescaling. If 0, rescale at each iteration. If a large value, never rescale. If < 0, rescale is set to 1.3. optimagic defaults to scipy's default. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_trust_constr .. code-block:: "scipy_trust_constr" Minimize a scalar function of one or more variables subject to constraints. .. warning:: In our benchmark using a quadratic objective function, the trust_constr algorithm did not find the optimum very precisely (less than 4 decimal places). If you require high precision, you should refine an optimum found with trust_constr with another local optimizer. .. note:: Its general nonlinear constraints' handling is not supported yet by optimagic. It switches between two implementations depending on the problem definition. It is the most versatile constrained minimization algorithm implemented in SciPy and the most appropriate for large-scale problems. For equality constrained problems it is an implementation of Byrd-Omojokun Trust-Region SQP method described in :cite:`Lalee1998` and in :cite:`Conn2000`, p. 549. When inequality constraints are imposed as well, it switches to the trust-region interior point method described in :cite:`Byrd1999`. This interior point algorithm in turn, solves inequality constraints by introducing slack variables and solving a sequence of equality-constrained barrier problems for progressively smaller values of the barrier parameter. The previously described equality constrained SQP method is used to solve the subproblems with increasing levels of accuracy as the iterate gets closer to a solution. It approximates the Hessian using the Broyden-Fletcher-Goldfarb-Shanno (BFGS) Hessian update strategy. - **convergence.gtol_abs** (float): Tolerance for termination by the norm of the Lagrangian gradient. The algorithm will terminate when both the infinity norm (i.e., max abs value) of the Lagrangian gradient and the constraint violation are smaller than the convergence.gtol_abs. For this algorithm we use scipy's gradient tolerance for trust_constr. This smaller tolerance is needed for the sum of squares tests to pass. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as convergence. - **convergence.xtol_rel** (float): Tolerance for termination by the change of the independent variable. The algorithm will terminate when the radius of the trust region used in the algorithm is smaller than the convergence.xtol_rel. - **trustregion.initial_radius** (float): Initial value of the trust region radius. The trust radius gives the maximum distance between solution points in consecutive iterations. It reflects the trust the algorithm puts in the local approximation of the optimization problem. For an accurate local approximation the trust-region should be large and for an approximation valid only close to the current point it should be a small one. The trust radius is automatically updated throughout the optimization process, with ``trustregion_initial_radius`` being its initial value. - **display** (bool): Set to True to print convergence messages. Default is False. SciPy name: **disp**. ``` ```{eval-rst} .. dropdown:: scipy_ls_dogbox .. code-block:: "scipy_ls_dogbox" Minimize a nonlinear least squares problem using a rectangular trust region method. Typical use case is small problems with bounds. Not recommended for problems with rank-deficient Jacobian. The algorithm supports the following options: - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is below this. - **convergence.gtol_rel** (float): Stop when the gradient, divided by the absolute value of the criterion function is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **tr_solver** (str): Method for solving trust-region subproblems, relevant only for 'trf' and 'dogbox' methods. - 'exact' is suitable for not very large problems with dense Jacobian matrices. The computational complexity per iteration is comparable to a singular value decomposition of the Jacobian matrix. - 'lsmr' is suitable for problems with sparse and large Jacobian matrices. It uses the iterative procedure `scipy.sparse.linalg.lsmr` for finding a solution of a linear least-squares problem and only requires matrix-vector product evaluations. If None (default), the solver is chosen based on the type of Jacobian returned on the first iteration. - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - ``tr_solver='exact'``: `tr_options` are ignored. - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. ``` ```{eval-rst} .. dropdown:: scipy_ls_trf .. code-block:: "scipy_ls_trf" Minimize a nonlinear least squares problem using a trustregion reflective method. Trust Region Reflective algorithm, particularly suitable for large sparse problems with bounds. Generally robust method. The algorithm supports the following options: - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is below this. - **convergence.gtol_rel** (float): Stop when the gradient, divided by the absolute value of the criterion function is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **tr_solver** (str): Method for solving trust-region subproblems, relevant only for 'trf' and 'dogbox' methods. - 'exact' is suitable for not very large problems with dense Jacobian matrices. The computational complexity per iteration is comparable to a singular value decomposition of the Jacobian matrix. - 'lsmr' is suitable for problems with sparse and large Jacobian matrices. It uses the iterative procedure `scipy.sparse.linalg.lsmr` for finding a solution of a linear least-squares problem and only requires matrix-vector product evaluations. If None (default), the solver is chosen based on the type of Jacobian returned on the first iteration. - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - ``tr_solver='exact'``: `tr_options` are ignored. - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. ``` ```{eval-rst} .. dropdown:: scipy_ls_lm .. code-block:: "scipy_ls_lm" Minimize a nonlinear least squares problem using a Levenberg-Marquardt method. Does not handle bounds and sparse Jacobians. Usually the most efficient method for small unconstrained problems. The algorithm supports the following options: - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is below this. - **convergence.gtol_rel** (float): Stop when the gradient, divided by the absolute value of the criterion function is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **tr_solver** (str): Method for solving trust-region subproblems, relevant only for 'trf' and 'dogbox' methods. - 'exact' is suitable for not very large problems with dense Jacobian matrices. The computational complexity per iteration is comparable to a singular value decomposition of the Jacobian matrix. - 'lsmr' is suitable for problems with sparse and large Jacobian matrices. It uses the iterative procedure `scipy.sparse.linalg.lsmr` for finding a solution of a linear least-squares problem and only requires matrix-vector product evaluations. If None (default), the solver is chosen based on the type of Jacobian returned on the first iteration. - **tr_solver_options** (dict): Keyword options passed to trust-region solver. - ``tr_solver='exact'``: `tr_options` are ignored. - ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`. ``` ```{eval-rst} .. dropdown:: scipy_basinhopping .. code-block:: "scipy_basinhopping" Find the global minimum of a function using the basin-hopping algorithm which combines a global stepping algorithm with local minimization at each step. Basin-hopping is a two-phase method that combines a global stepping algorithm with local minimization at each step. Designed to mimic the natural process of energy minimization of clusters of atoms, it works well for similar problems with “funnel-like, but rugged” energy landscapes. This is mainly supported for completeness. Consider optimagic's built in multistart optimization for a similar approach that can run multiple optimizations in parallel, supports all local algorithms in optimagic (as opposed to just those from scipy) and allows for a better visualization of the multistart history. When provided the derivative is passed to the local minimization method. The algorithm supports the following options: - **local_algorithm** (str/callable): Any scipy local minimizer: valid options are. "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA". "SLSQP". "trust-constr". "dogleg". "trust-ncg". "trust-exact". "trust-krylov". or a custom function for local minimization, default is "L-BFGS-B". - **n_local_optimizations**: (int) The number local optimizations. Default is 100 as in scipy's default. - **temperature**: (float) Controls the randomness in the optimization process. Higher the temperatures the larger jumps in function value will be accepted. Default is 1.0 as in scipy's default. - **stepsize**: (float) Maximum step size. Default is 0.5 as in scipy's default. - **local_algo_options**: (dict) Additional keyword arguments for the local minimizer. Check the documentation of the local scipy algorithms for details on what is supported. - **take_step**: (callable) Replaces the default step-taking routine. Default is None as in scipy's default. - **accept_test**: (callable) Define a test to judge the acception of steps. Default is None as in scipy's default. - **interval**: (int) Determined how often the step size is updated. Default is 50 as in scipy's default. - **convergence.n_unchanged_iterations**: (int) Number of iterations the global minimum estimate stays the same to stops the algorithm. Default is None as in scipy's default. - **seed**: (None, int, numpy.random.Generator,numpy.random.RandomState)Default is None as in scipy's default. - **target_accept_rate**: (float) Adjusts the step size. Default is 0.5 as in scipy's default. - **stepwise_factor**: (float) Step size multiplier upon each step. Lies between (0,1), default is 0.9 as in scipy's default. ``` ```{eval-rst} .. dropdown:: scipy_brute .. code-block:: "scipy_brute" Find the global minimum of a fuction over a given range by brute force. Brute force evaluates the criterion at each point and that is why better suited for problems with very few parameters. The start values are not actually used because the grid is only defined by bounds. It is still necessary for optimagic to infer the number and format of the parameters. Due to the parallelization, this algorithm cannot collect a history of parameters and criterion evaluations. The algorithm supports the following options: - **n_grid_points** (int): the number of grid points to use for the brute force search. Default is 20 as in scipy. - **polishing_function** (callable): Function to seek a more precise minimum near brute-force' best gridpoint taking brute-force's result at initial guess as a positional argument. Default is None providing no polishing. - **n_cores** (int): The number of cores on which the function is evaluated in parallel. Default 1. - **batch_evaluator** (str or callable). An optimagic batch evaluator. Default 'joblib'. ``` ```{eval-rst} .. dropdown:: scipy_differential_evolution .. code-block:: "scipy_differential_evolution" Find the global minimum of a multivariate function using differential evolution (DE). DE is a gradient-free method. Due to optimagic's general parameter format the integrality and vectorized arguments are not supported. The algorithm supports the following options: - **strategy** (str): Measure of quality to improve a candidate solution, can be one of the following keywords (default 'best1bin'.) - ‘best1bin’ - ‘best1exp’ - ‘rand1exp’ - ‘randtobest1exp’ - ‘currenttobest1exp’ - ‘best2exp’ - ‘rand2exp’ - ‘randtobest1bin’ - ‘currenttobest1bin’ - ‘best2bin’ - ‘rand2bin’ - ‘rand1bin’ - **stopping.maxiter** (int): The maximum number of criterion evaluations without polishing is(stopping.maxiter + 1) * population_size * number of parameters - **population_size_multiplier** (int): A multiplier setting the population size. The number of individuals in the population is population_size * number of parameters. The default 15. - **convergence.ftol_rel** (float): Default 0.01. - **mutation_constant** (float/tuple): The differential weight denoted by F in literature. Should be within 0 and 2. The tuple form is used to specify (min, max) dithering which can help speed convergence. Default is (0.5, 1). - **recombination_constant** (float): The crossover probability or CR in the literature determines the probability that two solution vectors will be combined to produce a new solution vector. Should be between 0 and 1. The default is 0.7. - **seed** (int): DE is stochastic. Define a seed for reproducability. - **polish** (bool): Uses scipy's L-BFGS-B for unconstrained problems and trust-constr for constrained problems to slightly improve the minimization. Default is True. - **sampling_method** (str/np.array): Specify the sampling method for the initial population. It can be one of the following options - "latinhypercube" - "sobol" - "halton" - "random" - an array specifying the initial population of shape (total population size, number of parameters). The initial population is clipped to bounds before use. Default is 'latinhypercube' - **convergence.ftol_abs** (float): CONVERGENCE_SECOND_BEST_ABSOLUTE_CRITERION_TOLERANCE - **n_cores** (int): The number of cores on which the function is evaluated in parallel. Default 1. - **batch_evaluator** (str or callable). An optimagic batch evaluator. Default 'joblib'. ``` ```{eval-rst} .. dropdown:: scipy_shgo .. code-block:: "scipy_shgo" Find the global minimum of a fuction using simplicial homology global optimization. The algorithm supports the following options: - **local_algorithm** (str): The local optimization algorithm to be used. Only COBYLA and SLSQP supports constraints. Valid options are "Nelder-Mead". "Powell". "CG". "BFGS". "Newton-CG". "L-BFGS-B". "TNC". "COBYLA". "SLSQP". "trust-constr". "dogleg". "trust-ncg". "trust-exact". "trust-krylov" or a custom function for local minimization, default is "L-BFGS-B". - **local_algo_options**: (dict) Additional keyword arguments for the local minimizer. Check the documentation of the local scipy algorithms for details on what is supported. - **n_sampling_points** (int): Specify the number of sampling points to construct the simplical complex. - **n_simplex_iterations** (int): Number of iterations to construct the simplical complex. Default is 1 as in scipy. - **sampling_method** (str/callable): The method to use for sampling the search space. Default 'simplicial'. - **max_sampling_evaluations** (int): The maximum number of evaluations of the criterion function in the sampling phase. - **convergence.minimum_criterion_value** (float): Specify the global minimum when it is known. Default is - np.inf. For maximization problems, flip the sign. - **convergence.minimum_criterion_tolerance** (float): Specify the relative error between the current best minimum and the supplied global criterion_minimum allowed. Default is scipy's default, 1e-4. - **stopping.maxiter** (int): The maximum number of iterations. - **stopping.maxfun** (int): The maximum number of criterion evaluations. - **stopping.max_processing_time** (int): The maximum time allowed for the optimization. - **minimum_homology_group_rank_differential** (int): The minimum difference in the rank of the homology group between iterations. - **symmetry** (bool): Specify whether the criterion contains symetric variables. - **minimize_every_iteration** (bool): Specify whether the gloabal sampling points are passed to the local algorithm in every iteration. - **max_local_minimizations_per_iteration** (int): The maximum number of local optimizations per iteration. Default False, i.e. no limit. - **infinity_constraints** (bool): Specify whether to save the sampling points outside the feasible domain. Default is True. ``` ```{eval-rst} .. dropdown:: scipy_dual_annealing .. code-block:: "scipy_dual_annealing" Find the global minimum of a function using dual annealing for continuous variables. The algorithm supports the following options: - **stopping.maxiter** (int): Specify the maximum number of global searh iterations. - **local_algorithm** (str): The local optimization algorithm to be used. valid options are: "Nelder-Mead", "Powell", "CG", "BFGS", "Newton-CG", "L-BFGS-B", "TNC", "COBYLA", "SLSQP", "trust-constr", "dogleg", "trust-ncg", "trust-exact", "trust-krylov", Default "L-BFGS-B". - **local_algo_options**: (dict) Additional keyword arguments for the local minimizer. Check the documentation of the local scipy algorithms for details on what is supported. - **initial_temperature** (float): The temparature algorithm starts with. The higher values lead to a wider search space. The range is (0.01, 5.e4] and default is 5230.0. - **restart_temperature_ratio** (float): Reanneling starts when the algorithm is decreased to initial_temperature * restart_temperature_ratio. Default is 2e-05. - **visit** (float): Specify the thickness of visiting distribution's tails. Range is (1, 3] and default is scipy's default, 2.62. - **accept** (float): Controls the probability of acceptance. Range is (-1e4, -5] and default is scipy's default, -5.0. Smaller values lead to lower acceptance probability. - **stopping.maxfun** (int): soft limit for the number of criterion evaluations. - **seed** (int, None or RNG): Dual annealing is a stochastic process. Seed or random number generator. Default None. - **no_local_search** (bool): Specify whether to apply a traditional Generalized Simulated Annealing with no local search. Default is False. ``` ```{eval-rst} .. dropdown:: scipy_direct .. code-block:: "scipy_direct" Find the global minimum of a function using dividing rectangles method. It is not necessary to provide an initial guess. The algorithm supports the following options: - **eps** (float): Specify the minimum difference of the criterion values between the current best hyperrectangle and the next potentially best hyperrectangle to be divided determining the trade off between global and local search. Default is 1e-6 differing from scipy's default 1e-4. - **stopping.maxfun** (int/None): Maximum number of criterion evaluations allowed. Default is None which caps the number of evaluations at 1000 * number of dimentions automatically. - **stopping.maxiter** (int): Maximum number of iterations allowed. - **locally_biased** (bool): Determine whether to use the locally biased variant of the algorithm DIRECT_L. Default is True. - **convergence.minimum_criterion_value** (float): Specify the global minimum when it is known. Default is minus infinity. For maximization problems, flip the sign. - **convergence.minimum_criterion_tolerance** (float): Specify the relative error between the current best minimum and the supplied global criterion_minimum allowed. Default is scipy's default, 1e-4. - **volume_hyperrectangle_tolerance** (float): Specify the smallest volume of the hyperrectangle containing the lowest criterion value allowed. Range is (0,1). Default is 1e-16. - **length_hyperrectangle_tolerance** (float): Depending on locally_biased it can refer to normalized side (True) or diagonal (False) length of the hyperrectangle containing the lowest criterion value. Range is (0,1). Default is scipy's default, 1e-6. ``` (own-algorithms)= ## Own optimizers We implement a few algorithms from scratch. They are currently considered experimental. ```{eval-rst} .. dropdown:: bhhh .. code-block:: "bhhh" Minimize a likelihood function using the BHHH algorithm. BHHH (:cite:`Berndt1974`) can - and should ONLY - be used for minimizing (or maximizing) a likelihood. It is similar to the Newton-Raphson algorithm, but replaces the Hessian matrix with the outer product of the gradient. This approximation is based on the information matrix equality (:cite:`Halbert1982`) and is thus only vaid when minimizing (or maximizing) a likelihood. The criterion function :func:`func` should return a dictionary with at least the entry ``{"contributions": array_or_pytree}`` where ``array_or_pytree`` contains the likelihood contributions of each individual. bhhh supports the following options: - **convergence.gtol_abs** (float): Stopping criterion for the gradient tolerance. Default is 1e-8. - **stopping.maxiter** (int): Maximum number of iterations. If reached, terminate. Default is 200. ``` ```{eval-rst} .. dropdown:: neldermead_parallel .. code-block:: "neldermead_parallel" Minimize a function using the neldermead_parallel algorithm. This is a parallel Nelder-Mead algorithm following Lee D., Wiswall M., A parallel implementation of the simplex function minimization routine, Computational Economics, 2007. The algorithm was implemented by Jacek Barszczewski The algorithm supports the following options: - **init_simplex_method** (string or callable): Name of the method to create initial simplex or callable which takes as an argument initial value of parameters and returns initial simplex as j+1 x j array, where j is length of x. The default is "gao_han". - **n_cores** (int): Degree of parallization. The default is 1 (no parallelization). - **adaptive** (bool): Adjust parameters of Nelder-Mead algorithm to account for simplex size. The default is True. - **stopping.maxiter** (int): Maximum number of algorithm iterations. The default is STOPPING_MAX_ITERATIONS. - **convergence.ftol_abs** (float): maximal difference between function value evaluated on simplex points. The default is CONVERGENCE_SECOND_BEST_ABSOLUTE_CRITERION_TOLERANCE. - **convergence.xtol_abs** (float): maximal distance between points in the simplex. The default is CONVERGENCE_SECOND_BEST_ABSOLUTE_PARAMS_TOLERANCE. - **batch_evaluator** (string or callable): See :ref:`batch_evaluators` for details. Default "joblib". ``` ```{eval-rst} .. dropdown:: pounders .. code-block:: "pounders" Minimize a function using the POUNDERS algorithm. POUNDERs (:cite:`Benson2017`, :cite:`Wild2015`, `GitHub repository `_) can be a useful tool for economists who estimate structural models using indirect inference, because unlike commonly used algorithms such as Nelder-Mead, POUNDERs is tailored for minimizing a non-linear sum of squares objective function, and therefore may require fewer iterations to arrive at a local optimum than Nelder-Mead. Scaling the problem is necessary such that bounds correspond to the unit hypercube :math:`[0, 1]^n`. For unconstrained problems, scale each parameter such that unit changes in parameters result in similar order-of-magnitude changes in the criterion value(s). pounders supports the following options: - **convergence.gtol_abs**: Convergence tolerance for the absolute gradient norm. Stop if norm of the gradient is less than this. Default is 1e-8. - **convergence.gtol_rel**: Convergence tolerance for the relative gradient norm. Stop if norm of the gradient relative to the criterion value is less than this. Default is 1-8. - **convergence.gtol_scaled**: Convergence tolerance for the scaled gradient norm. Stop if norm of the gradient divided by norm of the gradient at the initial parameters is less than this. Disabled, i.e. set to False, by default. - **max_interpolation_points** (int): Maximum number of interpolation points. Default is `2 * n + 1`, where `n` is the length of the parameter vector. - **stopping.maxiter** (int): Maximum number of iterations. If reached, terminate. Default is 2000. - **trustregion_initial_radius (float)**: Delta, initial trust-region radius. 0.1 by default. - **trustregion_minimal_radius** (float): Minimal trust-region radius. 1e-6 by default. - **trustregion_maximal_radius** (float): Maximal trust-region radius. 1e6 by default. - **trustregion_shrinking_factor_not_successful** (float): Shrinking factor of the trust-region radius in case the solution vector of the suproblem is not accepted, but the model is fully linear (i.e. "valid"). Defualt is 0.5. - **trustregion_expansion_factor_successful** (float): Shrinking factor of the trust-region radius in case the solution vector of the suproblem is accepted. Default is 2. - **theta1** (float): Threshold for adding the current x candidate to the model. Function argument to find_affine_points(). Default is 1e-5. - **theta2** (float): Threshold for adding the current x candidate to the model. Argument to get_interpolation_matrices_residual_model(). Default is 1e-4. - **trustregion_threshold_successful** (float): First threshold for accepting the solution vector of the subproblem as the best x candidate. Default is 0. - **trustregion_threshold_very_successful** (float): Second threshold for accepting the solution vector of the subproblem as the best x candidate. Default is 0.1. - **c1** (float): Treshold for accepting the norm of our current x candidate. Function argument to find_affine_points() for the case where input array *model_improving_points* is zero. - **c2** (int): Treshold for accepting the norm of our current x candidate. Equal to 10 by default. Argument to *find_affine_points()* in case the input array *model_improving_points* is not zero. - **trustregion_subproblem_solver** (str): Solver to use for the trust-region subproblem. Two internal solvers are supported: - "bntr": Bounded Newton Trust-Region (default, supports bound constraints) - "gqtpar": (does not support bound constraints) - **trustregion_subsolver_options** (dict): Options dictionary containing the stopping criteria for the subproblem. It takes different keys depending on the type of subproblem solver used. With the exception of the stopping criterion "maxiter", which is always included. If the subsolver "bntr" is used, the dictionary also contains the tolerance levels "gtol_abs", "gtol_rel", and "gtol_scaled". Moreover, the "conjugate_gradient_method" can be provided. Available conjugate gradient methods are: - "cg". In this case, two additional stopping criteria are "gtol_abs_cg" and "gtol_rel_cg" - "steihaug-toint" - "trsbox" (default) If the subsolver "gqtpar" is employed, the two stopping criteria are "k_easy" and "k_hard". None of the dictionary keys need to be specified by default, but can be. - **batch_evaluator** (str or callable): Name of a pre-implemented batch evaluator (currently "joblib" and "pathos_mp") or callable with the same interface as the optimagic batch_evaluators. Default is "joblib". - **n_cores (int)**: Number of processes used to parallelize the function evaluations. Default is 1. ``` (tao-algorithms)= ## Optimizers from the Toolkit for Advanced Optimization (TAO) We wrap the pounders algorithm from the Toolkit of Advanced optimization. To use it you need to have [petsc4py](https://pypi.org/project/petsc4py/) installed. ```{eval-rst} .. dropdown:: tao_pounders .. code-block:: "tao_pounders" Minimize a function using the POUNDERs algorithm. POUNDERs (:cite:`Benson2017`, :cite:`Wild2015`, `GitHub repository `_) can be a useful tool for economists who estimate structural models using indirect inference, because unlike commonly used algorithms such as Nelder-Mead, POUNDERs is tailored for minimizing a non-linear sum of squares objective function, and therefore may require fewer iterations to arrive at a local optimum than Nelder-Mead. Scaling the problem is necessary such that bounds correspond to the unit hypercube :math:`[0, 1]^n`. For unconstrained problems, scale each parameter such that unit changes in parameters result in similar order-of-magnitude changes in the criterion value(s). POUNDERs has several convergence criteria. Let :math:`X` be the current parameter vector, :math:`X_0` the initial parameter vector, :math:`g` the gradient, and :math:`f` the criterion function. ``absolute_gradient_tolerance`` stops the optimization if the norm of the gradient falls below :math:`\epsilon`. .. math:: ||g(X)|| < \epsilon ``relative_gradient_tolerance`` stops the optimization if the norm of the gradient relative to the criterion value falls below :math:`epsilon`. .. math:: \frac{||g(X)||}{|f(X)|} < \epsilon ``scaled_gradient_tolerance`` stops the optimization if the norm of the gradient is lower than some fraction :math:`epsilon` of the norm of the gradient at the initial parameters. .. math:: \frac{||g(X)||}{||g(X0)||} < \epsilon - **convergence.gtol_abs** (float): Stop if norm of gradient is less than this. If set to False the algorithm will not consider convergence.gtol_abs. - **convergence.gtol_rel** (float): Stop if relative norm of gradient is less than this. If set to False the algorithm will not consider convergence.gtol_rel. - **convergence.scaled_gradient_tolerance** (float): Stop if scaled norm of gradient is smaller than this. If set to False the algorithm will not consider convergence.scaled_gradient_tolerance. - **trustregion.initial_radius** (float): Initial value of the trust region radius. It must be :math:`> 0`. - **stopping.maxiter** (int): Alternative Stopping criterion. If set the routine will stop after the number of specified iterations or after the step size is sufficiently small. If the variable is set the default criteria will all be ignored. ``` (nag-algorithms)= ## Optimizers from the Numerical Algorithms Group (NAG) We wrap two algorithms from the numerical algorithms group. To use them, you need to install each of them separately: - `pip install DFO-LS` - `pip install Py-BOBYQA` ```{eval-rst} .. dropdown:: nag_dfols *Note*: We recommend to install `DFO-LS` version 1.5.3 or higher. Versions of 1.5.0 or lower also work but the versions `1.5.1` and `1.5.2` contain bugs that can lead to errors being raised. .. code-block:: "nag_dfols" Minimize a function with least squares structure using DFO-LS. The DFO-LS algorithm :cite:`Cartis2018b` is designed to solve the nonlinear least-squares minimization problem (with optional bound constraints). Remember to cite :cite:`Cartis2018b` when using DF-OLS in addition to optimagic. .. math:: \min_{x\in\mathbb{R}^n} &\quad f(x) := \sum_{i=1}^{m}r_{i}(x)^2 \\ \text{s.t.} &\quad \text{lower_bounds} \leq x \leq \text{upper_bounds} The :math:`r_{i}` are called root contributions in optimagic. DFO-LS is a derivative-free optimization algorithm, which means it does not require the user to provide the derivatives of f(x) or :math:`r_{i}(x)`, nor does it attempt to estimate them internally (by using finite differencing, for instance). There are two main situations when using a derivative-free algorithm (such as DFO-LS) is preferable to a derivative-based algorithm (which is the vast majority of least-squares solvers): 1. If the residuals are noisy, then calculating or even estimating their derivatives may be impossible (or at least very inaccurate). By noisy, we mean that if we evaluate :math:`r_{i}(x)` multiple times at the same value of x, we get different results. This may happen when a Monte Carlo simulation is used, for instance. 2. If the residuals are expensive to evaluate, then estimating derivatives (which requires n evaluations of each :math:`r_{i}(x)` for every point of interest x) may be prohibitively expensive. Derivative-free methods are designed to solve the problem with the fewest number of evaluations of the criterion as possible. To read the detailed documentation of the algorithm `click here `_. There are four possible convergence criteria: 1. when the lower trust region radius is shrunk below a minimum (``convergence.minimal_trustregion_radius_tolerance``). 2. when the improvements of iterations become very small (``convergence.slow_progress``). This is very similar to ``relative_criterion_tolerance`` but ``convergence.slow_progress`` is more general allowing to specify not only the threshold for convergence but also a period over which the improvements must have been very small. 3. when a sufficient reduction to the criterion value at the start parameters has been reached, i.e. when :math:`\frac{f(x)}{f(x_0)} \leq \text{convergence.ftol_scaled}` 4. when all evaluations on the interpolation points fall within a scaled version of the noise level of the criterion function. This is only applicable if the criterion function is noisy. You can specify this criterion with ``convergence.noise_corrected_criterion_tolerance``. DF-OLS supports resetting the optimization and doing a fast start by starting with a smaller interpolation set and growing it dynamically. For more information see `their detailed documentation `_ and :cite:`Cartis2018b`. - **clip_criterion_if_overflowing** (bool): see :ref:`algo_options`. convergence.minimal_trustregion_radius_tolerance (float): see :ref:`algo_options`. - **convergence.noise_corrected_criterion_tolerance** (float): Stop when the evaluations on the set of interpolation points all fall within this factor of the noise level. The default is 1, i.e. when all evaluations are within the noise level. If you want to not use this criterion but still flag your criterion function as noisy, set this tolerance to 0.0. .. warning:: Very small values, as in most other tolerances don't make sense here. - **convergence.ftol_scaled** (float): Terminate if a point is reached where the ratio of the criterion value to the criterion value at the start params is below this value, i.e. if :math:`f(x_k)/f(x_0) \leq \text{convergence.ftol_scaled}`. Note this is deactivated unless the lowest mathematically possible criterion value (0.0) is actually achieved. - **convergence.slow_progress** (dict): Arguments for converging when the evaluations over several iterations only yield small improvements on average, see see :ref:`algo_options` for details. - **initial_directions (str)**: see :ref:`algo_options`. - **interpolation_rounding_error** (float): see :ref:`algo_options`. - **noise_additive_level** (float): Used for determining the presence of noise and the convergence by all interpolation points being within noise level. 0 means no additive noise. Only multiplicative or additive is supported. - **noise_multiplicative_level** (float): Used for determining the presence of noise and the convergence by all interpolation points being within noise level. 0 means no multiplicative noise. Only multiplicative or additive is supported. - **noise_n_evals_per_point** (callable): How often to evaluate the criterion function at each point. This is only applicable for criterion functions with noise, when averaging multiple evaluations at the same point produces a more accurate value. The input parameters are the ``upper_trustregion_radius`` (:math:`\Delta`), the ``lower_trustregion_radius`` (:math:`\rho`), how many iterations the algorithm has been running for, ``n_iterations`` and how many resets have been performed, ``n_resets``. The function must return an integer. Default is no averaging (i.e. ``noise_n_evals_per_point(...) = 1``). - **random_directions_orthogonal** (bool): see :ref:`algo_options`. - **stopping.maxfun** (int): see :ref:`algo_options`. - **threshold_for_safety_step** (float): see :ref:`algo_options`. - **trustregion.expansion_factor_successful** (float): see :ref:`algo_options`. - **trustregion.expansion_factor_very_successful** (float): see :ref:`algo_options`. - **trustregion.fast_start_options** (dict): see :ref:`algo_options`. - **trustregion.initial_radius** (float): Initial value of the trust region radius. - **trustregion.method_to_replace_extra_points (str)**: If replacing extra points in successful iterations, whether to use geometry improving steps or the momentum method. Can be "geometry_improving" or "momentum". - **trustregion.n_extra_points_to_replace_successful** (int): The number of extra points (other than accepting the trust region step) to replace. Useful when ``trustregion.n_interpolation_points > len(x) + 1``. - **trustregion.n_interpolation_points** (int): The number of interpolation points to use. The default is :code:`len(x) + 1`. If using resets, this is the number of points to use in the first run of the solver, before any resets. - **trustregion.precondition_interpolation** (bool): see :ref:`algo_options`. - **trustregion.shrinking_factor_not_successful** (float): see :ref:`algo_options`. - **trustregion.shrinking_factor_lower_radius** (float): see :ref:`algo_options`. - **trustregion.shrinking_factor_upper_radius** (float): see :ref:`algo_options`. - **trustregion.threshold_successful** (float): Share of the predicted improvement that has to be achieved for a trust region iteration to count as successful. - **trustregion.threshold_very_successful** (float): Share of the predicted improvement that has to be achieved for a trust region iteration to count as very successful. ``` ```{eval-rst} .. dropdown:: nag_pybobyqa .. code-block:: "nag_pybobyqa" Minimize a function using the BOBYQA algorithm. BOBYQA (:cite:`Powell2009`, :cite:`Cartis2018`, :cite:`Cartis2018a`) is a derivative-free trust-region method. It is designed to solve nonlinear local minimization problems. Remember to cite :cite:`Powell2009` and :cite:`Cartis2018` when using pybobyqa in addition to optimagic. If you take advantage of the ``seek_global_optimum`` option, cite :cite:`Cartis2018a` additionally. There are two main situations when using a derivative-free algorithm like BOBYQA is preferable to derivative-based algorithms: 1. The criterion function is not deterministic, i.e. if we evaluate the criterion function multiple times at the same parameter vector we get different results. 2. The criterion function is very expensive to evaluate and only finite differences are available to calculate its derivative. The detailed documentation of the algorithm can be found `here `_. There are four possible convergence criteria: 1. when the trust region radius is shrunk below a minimum. This is approximately equivalent to an absolute parameter tolerance. 2. when the criterion value falls below an absolute, user-specified value, the optimization terminates successfully. 3. when insufficient improvements have been gained over a certain number of iterations. The (absolute) threshold for what constitutes an insufficient improvement, how many iterations have to be insufficient and with which iteration to compare can all be specified by the user. 4. when all evaluations on the interpolation points fall within a scaled version of the noise level of the criterion function. This is only applicable if the criterion function is noisy. - **clip_criterion_if_overflowing** (bool): see :ref:`algo_options`. - **convergence.criterion_value** (float): Terminate successfully if the criterion value falls below this threshold. This is deactivated (i.e. set to -inf) by default. - **convergence.minimal_trustregion_radius_tolerance** (float): Minimum allowed value of the trust region radius, which determines when a successful termination occurs. - **convergence.noise_corrected_criterion_tolerance** (float): Stop when the evaluations on the set of interpolation points all fall within this factor of the noise level. The default is 1, i.e. when all evaluations are within the noise level. If you want to not use this criterion but still flag your criterion function as noisy, set this tolerance to 0.0. .. warning:: Very small values, as in most other tolerances don't make sense here. - **convergence.slow_progress** (dict): Arguments for converging when the evaluations over several iterations only yield small improvements on average, see see :ref:`algo_options` for details. - **initial_directions** (str)``: see :ref:`algo_options`. - **interpolation_rounding_error** (float): see :ref:`algo_options`. - **noise_additive_level** (float): Used for determining the presence of noise and the convergence by all interpolation points being within noise level. 0 means no additive noise. Only multiplicative or additive is supported. - **noise_multiplicative_level** (float): Used for determining the presence of noise and the convergence by all interpolation points being within noise level. 0 means no multiplicative noise. Only multiplicative or additive is supported. - **noise_n_evals_per_point** (callable): How often to evaluate the criterion function at each point. This is only applicable for criterion functions with noise, when averaging multiple evaluations at the same point produces a more accurate value. The input parameters are the ``upper_trustregion_radius`` (``delta``), the ``lower_trustregion_radius`` (``rho``), how many iterations the algorithm has been running for, ``n_iterations`` and how many resets have been performed, ``n_resets``. The function must return an integer. Default is no averaging (i.e. ``noise_n_evals_per_point(...) = 1``). - **random_directions_orthogonal** (bool): see :ref:`algo_options`. - **seek_global_optimum** (bool): whether to apply the heuristic to escape local minima presented in :cite:`Cartis2018a`. Only applies for noisy criterion functions. - **stopping.maxfun** (int): see :ref:`algo_options`. - **threshold_for_safety_step** (float): see :ref:`algo_options`. - **trustregion.expansion_factor_successful** (float): see :ref:`algo_options`. - **trustregion.expansion_factor_very_successful** (float): see :ref:`algo_options`. - **trustregion.initial_radius** (float): Initial value of the trust region radius. - **trustregion.minimum_change_hession_for_underdetermined_interpolation** (bool): Whether to solve the underdetermined quadratic interpolation problem by minimizing the Frobenius norm of the Hessian, or change in Hessian. - **trustregion.n_interpolation_points** (int): The number of interpolation points to use. With $n=len(x)$ the default is $2n+1$ if the criterion is not noisy. Otherwise, it is set to $(n+1)(n+2)/2)$. Larger values are particularly useful for noisy problems. Py-BOBYQA requires .. math:: n + 1 \leq \text{trustregion.n_interpolation_points} \leq (n+1)(n+2)/2. - **trustregion.precondition_interpolation** (bool): see :ref:`algo_options`. - **trustregion.reset_options** (dict): Options for resetting the optimization, see :ref:`algo_options` for details. - **trustregion.shrinking_factor_not_successful** (float): see :ref:`algo_options`. - **trustregion.shrinking_factor_upper_radius** (float): see :ref:`algo_options`. - **trustregion.shrinking_factor_lower_radius** (float): see :ref:`algo_options`. - **trustregion.threshold_successful** (float): see :ref:`algo_options`. - **trustregion.threshold_very_successful** (float): see :ref:`algo_options`. ``` (pygmo-algorithms)= ## PYGMO2 Optimizers Please cite {cite}`Biscani2020` in addition to optimagic when using pygmo. optimagic supports the following [pygmo2](https://esa.github.io/pygmo2) optimizers. ```{eval-rst} .. dropdown:: pygmo_gaco .. code-block:: "pygmo_gaco" Minimize a scalar function using the generalized ant colony algorithm. The version available through pygmo is an generalized version of the original ant colony algorithm proposed by :cite:`Schlueter2009`. This algorithm can be applied to box-bounded problems. Ant colony optimization is a class of optimization algorithms modeled on the actions of an ant colony. Artificial "ants" (e.g. simulation agents) locate optimal solutions by moving through a parameter space representing all possible solutions. Real ants lay down pheromones directing each other to resources while exploring their environment. The simulated "ants" similarly record their positions and the quality of their solutions, so that in later simulation iterations more ants locate better solutions. The generalized ant colony algorithm generates future generations of ants by using a multi-kernel gaussian distribution based on three parameters (i.e., pheromone values) which are computed depending on the quality of each previous solution. The solutions are ranked through an oracle penalty method. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **batch_evaluator** (str or Callable): Name of a pre-implemented batch evaluator (currently 'joblib' and 'pathos_mp') or Callable with the same interface as the optimagic batch_evaluators. See :ref:`batch_evaluators`. - **n_cores** (int): Number of cores to use. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **kernel_size** (int): Number of solutions stored in the solution archive. - **speed_parameter_q** (float): This parameter manages the convergence speed towards the found minima (the smaller the faster). In the pygmo documentation it is referred to as $q$. It must be positive and can be larger than 1. The default is 1.0 until **threshold** is reached. Then it is set to 0.01. - **oracle** (float): oracle parameter used in the penalty method. - **accuracy** (float): accuracy parameter for maintaining a minimum penalty function's values distances. - **threshold** (int): when the iteration counter reaches the threshold the convergence speed is set to 0.01 automatically. To deactivate this effect set the threshold to stopping.maxiter which is the largest allowed value. - **speed_of_std_values_convergence** (int): parameter that determines the convergence speed of the standard deviations. This must be an integer (`n_gen_mark` in pygmo and pagmo). - **stopping.max_n_without_improvements** (int): if a positive integer is assigned here, the algorithm will count the runs without improvements, if this number exceeds the given value, the algorithm will be stopped. - **stopping.maxfun** (int): maximum number of function evaluations. - **focus** (float): this parameter makes the search for the optimum greedier and more focused on local improvements (the higher the greedier). If the value is very high, the search is more focused around the current best solutions. Values larger than 1 are allowed. - **cache** (bool): if True, memory is activated in the algorithm for multiple calls. ``` ```{eval-rst} .. dropdown:: pygmo_bee_colony .. code-block:: "pygmo_bee_colony" Minimize a scalar function using the artifical bee colony algorithm. The Artificial Bee Colony Algorithm was originally proposed by :cite:`Karaboga2007`. The implemented version of the algorithm is proposed in :cite:`Mernik2015`. The algorithm is only suited for bounded parameter spaces. - **stopping.maxiter** (int): Number of generations to evolve. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **max_n_trials** (int): Maximum number of trials for abandoning a source. Default is 1. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 20. ``` ```{eval-rst} .. dropdown:: pygmo_de .. code-block:: "pygmo_de" Minimize a scalar function using the differential evolution algorithm. Differential Evolution is a heuristic optimizer originally presented in :cite:`Storn1997`. The algorithm is only suited for bounded parameter spaces. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 10. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **weight_coefficient** (float): Weight coefficient. It is denoted by $F$ in the main paper and must lie in [0, 2]. It controls the amplification of the differential variation $(x_{r_2, G} - x_{r_3, G})$. - **crossover_probability** (float): Crossover probability. - **mutation_variant (str or int)**: code for the mutation variant to create a new candidate individual. The default is . The following are available: - "best/1/exp" (1, when specified as int) - "rand/1/exp" (2, when specified as int) - "rand-to-best/1/exp" (3, when specified as int) - "best/2/exp" (4, when specified as int) - "rand/2/exp" (5, when specified as int) - "best/1/bin" (6, when specified as int) - "rand/1/bin" (7, when specified as int) - "rand-to-best/1/bin" (8, when specified as int) - "best/2/bin" (9, when specified as int) - "rand/2/bin" (10, when specified as int) - **convergence.criterion_tolerance**: stopping criteria on the criterion tolerance. Default is 1e-6. It is not clear whether this is the absolute or relative criterion tolerance. - **convergence.xtol_rel**: stopping criteria on the x tolerance. In pygmo the default is 1e-6 but we use our default value of 1e-5. ``` ```{eval-rst} .. dropdown:: pygmo_sea .. code-block:: "pygmo_sea" Minimize a scalar function using the (N+1)-ES simple evolutionary algorithm. This algorithm represents the simplest evolutionary strategy, where a population of $\lambda$ individuals at each generation produces one offspring by mutating its best individual uniformly at random within the bounds. Should the offspring be better than the worst individual in the population it will substitute it. See :cite:`Oliveto2007`. The algorithm is only suited for bounded parameter spaces. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 10. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): number of generations to consider. Each generation will compute the objective function once. ``` ```{eval-rst} .. dropdown:: pygmo_sga .. code-block:: "pygmo_sga" Minimize a scalar function using a simple genetic algorithm. A detailed description of the algorithm can be found `in the pagmo2 documentation `_. See also :cite:`Oliveto2007`. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **crossover_probability** (float): Crossover probability. - **crossover_strategy** (str): the crossover strategy. One of “exponential”,“binomial”, “single” or “sbx”. Default is "exponential". - **eta_c** (float): distribution index for “sbx” crossover. This is an inactive parameter if other types of crossovers are selected. Can be in [1, 100]. - **mutation_probability** (float): Mutation probability. - **mutation_strategy** (str): Mutation strategy. Must be "gaussian", "polynomial" or "uniform". Default is "polynomial". - **mutation_polynomial_distribution_index** (float): Must be in [0, 1]. Default is 1. - **mutation_gaussian_width** (float): Must be in [0, 1]. Default is 1. - **selection_strategy (str)**: Selection strategy. Must be "tournament" or "truncated". - **selection_truncated_n_best** (int): number of best individuals to use in the "truncated" selection mechanism. - **selection_tournament_size** (int): size of the tournament in the "tournament" selection mechanism. Default is 1. ``` ```{eval-rst} .. dropdown:: pygmo_sade .. code-block:: "pygmo_sade" Minimize a scalar function using Self-adaptive Differential Evolution. The original Differential Evolution algorithm (pygmo_de) can be significantly improved introducing the idea of parameter self-adaptation. Many different proposals have been made to self-adapt both the crossover and the F parameters of the original differential evolution algorithm. pygmo's implementation supports two different mechanisms. The first one, proposed by :cite:`Brest2006`, does not make use of the differential evolution operators to produce new values for the weight coefficient $F$ and the crossover probability $CR$ and, strictly speaking, is thus not self-adaptation, rather parameter control. The resulting differential evolution variant is often referred to as jDE. The second variant is inspired by the ideas introduced by :cite:`Elsayed2011` and uses a variaton of the selected DE operator to produce new $CR$ anf $F$ parameters for each individual. This variant is referred to iDE. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - jde (bool): Whether to use the jDE self-adaptation variant to control the $F$ and $CR$ parameter. If True jDE is used, else iDE. - **stopping.maxiter** (int): Number of generations to evolve. - **mutation_variant** (int or str): code for the mutation variant to create a new candidate individual. The default is "rand/1/exp". The first ten are the classical mutation variants introduced in the orginal DE algorithm, the remaining ones are, instead, considered in the work by :cite:`Elsayed2011`. The following are available: - "best/1/exp" or 1 - "rand/1/exp" or 2 - "rand-to-best/1/exp" or 3 - "best/2/exp" or 4 - "rand/2/exp" or 5 - "best/1/bin" or 6 - "rand/1/bin" or 7 - "rand-to-best/1/bin" or 8 - "best/2/bin" or 9 - "rand/2/bin" or 10 - "rand/3/exp" or 11 - "rand/3/bin" or 12 - "best/3/exp" or 13 - "best/3/bin" or 14 - "rand-to-current/2/exp" or 15 - "rand-to-current/2/bin" or 16 - "rand-to-best-and-current/2/exp" or 17 - "rand-to-best-and-current/2/bin" or 18 - **keep_adapted_params** (bool): when true the adapted parameters $CR$ anf $F$ are not reset between successive calls to the evolve method. Default is False. - ftol (float): stopping criteria on the x tolerance. - xtol (float): stopping criteria on the f tolerance. ``` ```{eval-rst} .. dropdown:: pygmo_cmaes .. code-block:: "pygmo_cmaes" Minimize a scalar function using the Covariance Matrix Evolutionary Strategy. CMA-ES is one of the most successful algorithm, classified as an Evolutionary Strategy, for derivative-free global optimization. The version supported by optimagic is the version described in :cite:`Hansen2006`. In contrast to the pygmo version, optimagic always sets force_bounds to True. This avoids that ill defined parameter values are evaluated. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **backward_horizon** (float): backward time horizon for the evolution path. It must lie betwen 0 and 1. - **variance_loss_compensation** (float): makes partly up for the small variance loss in case the indicator is zero. `cs` in the MATLAB Code of :cite:`Hansen2006`. It must lie between 0 and 1. - **learning_rate_rank_one_update** (float): learning rate for the rank-one update of the covariance matrix. `c1` in the pygmo and pagmo documentation. It must lie between 0 and 1. - **learning_rate_rank_mu_update** (float): learning rate for the rank-mu update of the covariance matrix. `cmu` in the pygmo and pagmo documentation. It must lie between 0 and 1. - **initial_step_size** (float): initial step size, :math:`\sigma^0` in the original paper. - **ftol** (float): stopping criteria on the x tolerance. - **xtol** (float): stopping criteria on the f tolerance. - **keep_adapted_params** (bool): when true the adapted parameters are not reset between successive calls to the evolve method. Default is False. ``` ```{eval-rst} .. dropdown:: pygmo_simulated_annealing .. code-block:: "pygmo_simulated_annealing" Minimize a function with the simulated annealing algorithm. This version of the simulated annealing algorithm is, essentially, an iterative random search procedure with adaptive moves along the coordinate directions. It permits uphill moves under the control of metropolis criterion, in the hope to avoid the first local minima encountered. This version is the one proposed in :cite:`Corana1987`. .. note: When selecting the starting and final temperature values it helps to think about the tempertaure as the deterioration in the objective function value that still has a 37% chance of being accepted. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **start_temperature** (float): starting temperature. Must be > 0. - **end_temperature** (float): final temperature. Our default (0.01) is lower than in pygmo and pagmo. The final temperature must be positive. - **n_temp_adjustments** (int): number of temperature adjustments in the annealing schedule. - **n_range_adjustments** (int): number of adjustments of the search range performed at a constant temperature. - **bin_size** (int): number of mutations that are used to compute the acceptance rate. - **start_range** (float): starting range for mutating the decision vector. It must lie between 0 and 1. ``` ```{eval-rst} .. dropdown:: pygmo_pso .. code-block:: "pygmo_pso" Minimize a scalar function using Particle Swarm Optimization. Particle swarm optimization (PSO) is a population based algorithm inspired by the foraging behaviour of swarms. In PSO each point has memory of the position where it achieved the best performance xli (local memory) and of the best decision vector :math:`x^g` in a certain neighbourhood, and uses this information to update its position. For a survey on particle swarm optimization algorithms, see :cite:`Poli2007`. Each particle determines its future position :math:`x_{i+1} = x_i + v_i` where .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - x^{l}_i) + \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 10. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **omega** (float): depending on the variant chosen, :math:`\omega` is the particles' inertia weight or the construction coefficient. It must lie between 0 and 1. - **force_of_previous_best** (float): :math:`\eta_1` in the equation above. It's the magnitude of the force, applied to the particle’s velocity, in the direction of its previous best position. It must lie between 0 and 4. - **force_of_best_in_neighborhood** (float): :math:`\eta_2` in the equation above. It's the magnitude of the force, applied to the particle’s velocity, in the direction of the best position in its neighborhood. It must lie between 0 and 4. - **max_velocity** (float): maximum allowed particle velocity as fraction of the box bounds. It must lie between 0 and 1. - **algo_variant (int or str)**: algorithm variant to be used: - 1 or "canonical_inertia": Canonical (with inertia weight) - 2 or "social_and_cog_rand": Same social and cognitive rand. - 3 or "all_components_rand": Same rand. for all components - 4 or "one_rand": Only one rand. - 5 or "canonical_constriction": Canonical (with constriction fact.) - 6 or "fips": Fully Informed (FIPS) - **neighbor_definition (int or str)**: swarm topology that defines each particle's neighbors that is to be used: - 1 or "gbest" - 2 or "lbest" - 3 or "Von Neumann" - 4 or "Adaptive random" - **neighbor_param** (int): the neighbourhood parameter. If the lbest topology is selected (neighbor_definition=2), it represents each particle's indegree (also outdegree) in the swarm topology. Particles have neighbours up to a radius of k = neighbor_param / 2 in the ring. If the Randomly-varying neighbourhood topology is selected (neighbor_definition=4), it represents each particle’s maximum outdegree in the swarm topology. The minimum outdegree is 1 (the particle always connects back to itself). If neighbor_definition is 1 or 3 this parameter is ignored. - **keep_velocities** (bool): when true the particle velocities are not reset between successive calls to `evolve`. ``` ```{eval-rst} .. dropdown:: pygmo_pso_gen .. code-block:: "pygmo_pso_gen" Minimize a scalar function with generational Particle Swarm Optimization. Particle Swarm Optimization (generational) is identical to pso, but does update the velocities of each particle before new particle positions are computed (taking into consideration all updated particle velocities). Each particle is thus evaluated on the same seed within a generation as opposed to the standard PSO which evaluates single particle at a time. Consequently, the generational PSO algorithm is suited for stochastic optimization problems. For a survey on particle swarm optimization algorithms, see :cite:`Poli2007`. Each particle determines its future position :math:`x_{i+1} = x_i + v_i` where .. math:: v_{i+1} = \omega (v_i + \eta_1 \cdot \mathbf{r}_1 \cdot (x_i - x^{l}_i) + \eta_2 \cdot \mathbf{r}_2 \cdot (x_i - x^g)) - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 10. - **batch_evaluator (str or Callable)**: Name of a pre-implemented batch evaluator (currently 'joblib' and 'pathos_mp') or Callable with the same interface as the optimagic batch_evaluators. See :ref:`batch_evaluators`. - **n_cores** (int): Number of cores to use. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **omega** (float): depending on the variant chosen, :math:`\omega` is the particles' inertia weight or the constructuion coefficient. It must lie between 0 and 1. - **force_of_previous_best** (float): :math:`\eta_1` in the equation above. It's the magnitude of the force, applied to the particle’s velocity, in the direction of its previous best position. It must lie between 0 and 4. - **force_of_best_in_neighborhood** (float): :math:`\eta_2` in the equation above. It's the magnitude of the force, applied to the particle’s velocity, in the direction of the best position in its neighborhood. It must lie between 0 and 4. - **max_velocity** (float): maximum allowed particle velocity as fraction of the box bounds. It must lie between 0 and 1. - **algo_variant** (int): code of the algorithm's variant to be used: - 1 or "canonical_inertia": Canonical (with inertia weight) - 2 or "social_and_cog_rand": Same social and cognitive rand. - 3 or "all_components_rand": Same rand. for all components - 4 or "one_rand": Only one rand. - 5 or "canonical_constriction": Canonical (with constriction fact.) - 6 or "fips": Fully Informed (FIPS) - **neighbor_definition** (int): code for the swarm topology that defines each particle's neighbors that is to be used: - 1 or "gbest" - 2 or "lbest" - 3 or "Von Neumann" - 4 or "Adaptive random" - **neighbor_param** (int): the neighbourhood parameter. If the lbest topology is selected (neighbor_definition=2), it represents each particle's indegree (also outdegree) in the swarm topology. Particles have neighbours up to a radius of k = neighbor_param / 2 in the ring. If the Randomly-varying neighbourhood topology is selected (neighbor_definition=4), it represents each particle’s maximum outdegree in the swarm topology. The minimum outdegree is 1 (the particle always connects back to itself). If neighbor_definition is 1 or 3 this parameter is ignored. - **keep_velocities** (bool): when true the particle velocities are not reset between successive calls to `evolve`. ``` ```{eval-rst} .. dropdown:: pygmo_mbh .. code-block:: "pygmo_mbh" Minimize a scalar function using generalized Monotonic Basin Hopping. Monotonic basin hopping, or simply, basin hopping, is an algorithm rooted in the idea of mapping the objective function $f(x_0)$ into the local minima found starting from $x_0$. This simple idea allows a substantial increase of efficiency in solving problems, such as the Lennard-Jones cluster or the MGA-1DSM interplanetary trajectory problem that are conjectured to have a so-called funnel structure. See :cite:`Wales1997` for the paper introducing the basin hopping idea for a Lennard-Jones cluster optimization. pygmo provides an original generalization of this concept resulting in a meta-algorithm that operates on a population. When a population containing a single individual is used the original method is recovered. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 250. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **inner_algorithm** (pygmo.algorithm): an pygmo algorithm or a user-defined algorithm, either C++ or Python. If None the `pygmo.compass_search` algorithm will be used. - **stopping.max_inner_runs_without_improvement** (int): consecutive runs of the inner algorithm that need to result in no improvement for mbh to stop. - **perturbation** (float): the perturbation to be applied to each component. ``` ```{eval-rst} .. dropdown:: pygmo_xnes .. code-block:: "pygmo_xnes" Minimize a scalar function using Exponential Evolution Strategies. Exponential Natural Evolution Strategies is an algorithm closely related to CMAES and based on the adaptation of a gaussian sampling distribution via the so-called natural gradient. Like CMAES it is based on the idea of sampling new trial vectors from a multivariate distribution and using the new sampled points to update the distribution parameters. Naively this could be done following the gradient of the expected fitness as approximated by a finite number of sampled points. While this idea offers a powerful lead on algorithmic construction it has some major drawbacks that are solved in the so-called Natural Evolution Strategies class of algorithms by adopting, instead, the natural gradient. xNES is one of the most performing variants in this class. See :cite:`Glasmachers2010` and the `pagmo documentation on xNES `_ for details. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **learning_rate_mean_update** (float): learning rate for the mean update (:math:`\eta_\mu`). It must be between 0 and 1 or None. - **learning_rate_step_size_update** (float): learning rate for the step-size update. It must be between 0 and 1 or None. - **learning_rate_cov_matrix_update** (float): learning rate for the covariance matrix update. It must be between 0 and 1 or None. - **initial_search_share** (float): share of the given search space that will be initally searched. It must be between 0 and 1. Default is 1. - **ftol** (float): stopping criteria on the x tolerance. - **xtol** (float): stopping criteria on the f tolerance. - **keep_adapted_params** (bool): when true the adapted parameters are not reset between successive calls to the evolve method. Default is False. ``` ```{eval-rst} .. dropdown:: pygmo_gwo .. code-block:: "pygmo_gwo" Minimize a scalar function usinng the Grey Wolf Optimizer. The grey wolf optimizer was proposed by :cite:`Mirjalili2014`. The pygmo implementation that is wrapped by optimagic is pased on the pseudo code provided in that paper. This algorithm is a classic example of a highly criticizable line of search that led in the first decades of our millenia to the development of an entire zoo of metaphors inspiring optimzation heuristics. In our opinion they, as is the case for the grey wolf optimizer, are often but small variations of already existing heuristics rebranded with unnecessray and convoluted biological metaphors. In the case of GWO this is particularly evident as the position update rule is shokingly trivial and can also be easily seen as a product of an evolutionary metaphor or a particle swarm one. Such an update rule is also not particulary effective and results in a rather poor performance most of times. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. ``` ```{eval-rst} .. dropdown:: pygmo_compass_search .. code-block:: "pygmo_compass_search" Minimize a scalar function using compass search. The algorithm is described in :cite:`Kolda2003`. It is considered slow but reliable. It should not be used for stochastic problems. - **population_size** (int): Size of the population. Even though the algorithm is not population based the population size does affect the results of the algorithm. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxfun** (int): maximum number of function evaluations. - **start_range** (float): the start range. Must be in (0, 1]. - **stop_range** (float): the stop range. Must be in (0, start_range]. - **reduction_coeff** (float): the range reduction coefficient. Must be in (0, 1). ``` ```{eval-rst} .. dropdown:: pygmo_ihs .. code-block:: "pygmo_ihs" Minimize a scalar function using the improved harmony search algorithm. Improved harmony search (IHS) was introduced by :cite:`Mahdavi2007`. IHS supports stochastic problems. - **population_size** (int): Size of the population. If None, it's twice the number of parameters. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **stopping.maxiter** (int): Number of generations to evolve. - **choose_from_memory_probability** (float): probability of choosing from memory (similar to a crossover probability). - **min_pitch_adjustment_rate** (float): minimum pitch adjustment rate. (similar to a mutation rate). It must be between 0 and 1. - **max_pitch_adjustment_rate** (float): maximum pitch adjustment rate. (similar to a mutation rate). It must be between 0 and 1. - **min_distance_bandwidth** (float): minimum distance bandwidth. (similar to a mutation width). It must be positive. - **max_distance_bandwidth** (float): maximum distance bandwidth. (similar to a mutation width). ``` ```{eval-rst} .. dropdown:: pygmo_de1220 .. code-block:: "pygmo_de1220" Minimize a scalar function using Self-adaptive Differential Evolution, pygmo flavor. See `the PAGMO documentation for details `_. - **population_size** (int): Size of the population. If None, it's twice the number of parameters but at least 64. - **seed** (int): seed used by the internal random number generator. - **discard_start_params** (bool): If True, the start params are not guaranteed to be part of the initial population. This saves one criterion function evaluation that cannot be done in parallel with other evaluations. Default False. - **jde** (bool): Whether to use the jDE self-adaptation variant to control the $F$ and $CR$ parameter. If True jDE is used, else iDE. - **stopping.maxiter** (int): Number of generations to evolve. - **allowed_variants** (array-like object): allowed mutation variants (can be codes or strings). Each code refers to one mutation variant to create a new candidate individual. The first ten refer to the classical mutation variants introduced in the original DE algorithm, the remaining ones are, instead, considered in the work by :cite:`Elsayed2011`. The default is ["rand/1/exp", "rand-to-best/1/exp", "rand/1/bin", "rand/2/bin", "best/3/exp", "best/3/bin", "rand-to-current/2/exp", "rand-to-current/2/bin"]. The following are available: - 1 or "best/1/exp" - 2 or "rand/1/exp" - 3 or "rand-to-best/1/exp" - 4 or "best/2/exp" - 5 or "rand/2/exp" - 6 or "best/1/bin" - 7 or "rand/1/bin" - 8 or "rand-to-best/1/bin" - 9 or "best/2/bin" - 10 or "rand/2/bin" - 11 or "rand/3/exp" - 12 or "rand/3/bin" - 13 or "best/3/exp" - 14 or "best/3/bin" - 15 or "rand-to-current/2/exp" - 16 or "rand-to-current/2/bin" - 17 or "rand-to-best-and-current/2/exp" - 18 or "rand-to-best-and-current/2/bin" - **keep_adapted_params** (bool): when true the adapted parameters $CR$ anf $F$ are not reset between successive calls to the evolve method. Default is False. - **ftol** (float): stopping criteria on the x tolerance. - **xtol** (float): stopping criteria on the f tolerance. ``` (ipopt-algorithm)= ## The Interior Point Optimizer (ipopt) optimagic's support for the Interior Point Optimizer ({cite}`Waechter2005`, {cite}`Waechter2005a`, {cite}`Waechter2005b`, {cite}`Nocedal2009`) is built on [cyipopt](https://cyipopt.readthedocs.io/en/latest/index.html), a Python wrapper for the [Ipopt optimization package](https://coin-or.github.io/Ipopt/index.html). To use ipopt, you need to have [cyipopt installed](https://cyipopt.readthedocs.io/en/latest/index.html) (`conda install cyipopt`). ```{eval-rst} .. dropdown:: ipopt .. code-block:: "ipopt" Minimize a scalar function using the Interior Point Optimizer. This implementation of the Interior Point Optimizer (:cite:`Waechter2005`, :cite:`Waechter2005a`, :cite:`Waechter2005b`, :cite:`Nocedal2009`) relies on `cyipopt `_, a Python wrapper for the `Ipopt optimization package `_. There are two levels of termination criteria. If the usual "desired" tolerances (see tol, dual_inf_tol etc) are satisfied at an iteration, the algorithm immediately terminates with a success message. On the other hand, if the algorithm encounters "acceptable_iter" many iterations in a row that are considered "acceptable", it will terminate before the desired convergence tolerance is met. This is useful in cases where the algorithm might not be able to achieve the "desired" level of accuracy. The options are analogous to the ones in the `ipopt documentation `_ with the exception of the linear solver options which are here bundled into a dictionary. Any argument that takes "yes" and "no" in the ipopt documentation can also be passed as a `True` and `False`, respectively. and any option that accepts "none" in ipopt accepts a Python `None`. The following options are not supported: - `num_linear_variables`: since optimagic may reparametrize your problem and this changes the parameter problem, we do not support this option. - derivative checks - print options. - **convergence.ftol_rel** (float): The algorithm terminates successfully, if the (scaled) non linear programming error becomes smaller than this value. - **mu_target** (float): Desired value of complementarity. Usually, the barrier parameter is driven to zero and the termination test for complementarity is measured with respect to zero complementarity. However, in some cases it might be desired to have Ipopt solve barrier problem for strictly positive value of the barrier parameter. In this case, the value of "mu_target" specifies the final value of the barrier parameter, and the termination tests are then defined with respect to the barrier problem for this value of the barrier parameter. The valid range for this real option is 0 ≤ mu_target and its default value is 0. - **s_max** (float): Scaling threshold for the NLP error. - **stopping.maxiter** (int): If the maximum number of iterations is reached, the optimization stops, but we do not count this as successful convergence. The difference to ``max_criterion_evaluations`` is that one iteration might need several criterion evaluations, for example in a line search or to determine if the trust region radius has to be shrunk. - **stopping.max_wall_time_seconds** (float): Maximum number of walltime clock seconds. - **stopping.max_cpu_time** (float): Maximum number of CPU seconds. A limit on CPU seconds that Ipopt can use to solve one problem. If during the convergence check this limit is exceeded, Ipopt will terminate with a corresponding message. The valid range for this real option is 0 < max_cpu_time and its default value is :math:`1e+20` . - **dual_inf_tol** (float): Desired threshold for the dual infeasibility. Absolute tolerance on the dual infeasibility. Successful termination requires that the max-norm of the (unscaled) dual infeasibility is less than this threshold. The valid range for this real option is 0 < dual_inf_tol and its default value is 1. - **constr_viol_tol** (float): Desired threshold for the constraint and bound violation. Absolute tolerance on the constraint and variable bound violation. Successful termination requires that the max-norm of the (unscaled) constraint violation is less than this threshold. If option ``bound_relax_factor`` is not zero 0, then Ipopt relaxes given variable bounds. The value of constr_viol_tol is used to restrict the absolute amount of this bound relaxation. The valid range for this real option is 0 < constr_viol_tol and its default value is 0.0001. - **compl_inf_tol** (float): Desired threshold for the complementarity conditions. Absolute tolerance on the complementarity. Successful termination requires that the max-norm of the (unscaled) complementarity is less than this threshold. The valid range for this real option is 0 < text{compl_inf_tol and its default is 0.0001. - **acceptable_iter** (int): Number of "acceptable" iterates before termination. If the algorithm encounters this many successive "acceptable" iterates (see above on the acceptable heuristic), it terminates, assuming that the problem has been solved to best possible accuracy given round-off. If it is set to zero, this heuristic is disabled. The valid range for this integer option is 0 ≤ acceptable_iter. - **acceptable_tol** (float):"Acceptable" convergence tolerance (relative). Determines which (scaled) overall optimality error is considered to be "acceptable". The valid range for this real option is 0 < acceptable_tol. - **acceptable_dual_inf_tol** (float): "Acceptance" threshold for the dual infeasibility. Absolute tolerance on the dual infeasibility. "Acceptable" termination requires that the (max-norm of the unscaled) dual infeasibility is less than this threshold; see also ``acceptable_tol`` . The valid range for this real option is 0 < acceptable_dual_inf_tol and its default value is :math:`1e+10.` - **acceptable_constr_viol_tol** (float): "Acceptance" threshold for the constraint violation. Absolute tolerance on the constraint violation. "Acceptable" termination requires that the max-norm of the (unscaled) constraint violation is less than this threshold; see also ``acceptable_tol`` . The valid range for this real option is 0 < acceptable_constr_viol_tol and its default value is 0.01. - **acceptable_compl_inf_tol** (float): "Acceptance" threshold for the complementarity conditions. Absolute tolerance on the complementarity. "Acceptable" termination requires that the max-norm of the (unscaled) complementarity is less than this threshold; see also ``acceptable_tol`` . The valid range for this real option is 0 < text{acceptable_compl_inf_tol and its default value is 0.01. - **acceptable_obj_change_tol** (float): "Acceptance" stopping criterion based on objective function change. If the relative change of the objective function (scaled by :math:`max(1,|f(x)|)` ) is less than this value, this part of the acceptable tolerance termination is satisfied; see also ``acceptable_tol`` . This is useful for the quasi-Newton option, which has trouble to bring down the dual infeasibility. The valid range for this real option is 0 ≤ acceptable_obj_change_tol and its default value is :math:`1e+20` . - **diverging_iterates_tol** (float): Threshold for maximal value of primal iterates. If any component of the primal iterates exceeded this value (in absolute terms), the optimization is aborted with the exit message that the iterates seem to be diverging. The valid range for this real option is 0 < diverging_iterates_tol and its default value is :math:`1e+20` . - **nlp_lower_bound_inf** (float): any bound less or equal this value will be considered -inf (i.e. not lwer bounded). The valid range for this real option is unrestricted and its default value is :math:`-1e+19` . - **nlp_upper_bound_inf** (float): any bound greater or this value will be considered :math:`+\inf` (i.e. not upper bunded). The valid range for this real option is unrestricted and its default value is :math:`1e+19` . - **fixed_variable_treatment (str)**: Determines how fixed variables should be handled. The main difference between those options is that the starting point in the "make_constraint" case still has the fixed variables at their given values, whereas in the case "make_parameter(_nodual)" the functions are always evaluated with the fixed values for those variables. Also, for "relax_bounds", the fixing bound constraints are relaxed (according to ``bound_relax_factor`` ). For all but "make_parameter_nodual", bound multipliers are computed for the fixed variables. The default value for this string option is "make_parameter". Possible values: - "make_parameter": Remove fixed variable from optimization variables - "make_parameter_nodual": Remove fixed variable from optimization variables and do not compute bound multipliers for fixed variables - "make_constraint": Add equality constraints fixing variables - "relax_bounds": Relax fixing bound constraints - **dependency_detector (str)**: Indicates which linear solver should be used to detect linearly dependent equality constraints. This is experimental and does not work well. The default value for this string option is "none". Possible values: - "none" or None: don't check; no extra work at beginning - "mumps": use MUMPS - "wsmp": use WSMP - "ma28": use MA28 - **dependency_detection_with_rhs (str or bool)**: Indicates if the right hand sides of the constraints should be considered in addition to gradients during dependency detection. The default value for this string option is "no". Possible values: 'yes', 'no', True, False. - **kappa_d** (float): Weight for linear damping term (to handle one-sided bounds). See Section 3.7 in implementation paper. The valid range for this real option is 0 ≤ kappa_d and its default value is :math:`1e-05` . - **bound_relax_factor** (float): Factor for initial relaxation of the bounds. Before start of the optimization, the bounds given by the user are relaxed. This option sets the factor for this relaxation. Additional, the constraint violation tolerance ``constr_viol_tol`` is used to bound the relaxation by an absolute value. If it is set to zero, then then bounds relaxation is disabled. See Eqn.(35) in implementation paper. Note that the constraint violation reported by Ipopt at the end of the solution process does not include violations of the original (non-relaxed) variable bounds. See also option honor_original_bounds. The valid range for this real option is 0 ≤ bound_relax_factor and its default value is :math:`1e-08` . - **honor_original_bounds** (str or bool): Indicates whether final points should be projected into original bunds. Ipopt might relax the bounds during the optimization (see, e.g., option ``bound_relax_factor`` ). This option determines whether the final point should be projected back into the user-provide original bounds after the optimization. Note that violations of constraints and complementarity reported by Ipopt at the end of the solution process are for the non-projected point. The default value for this string option is "no". Possible values: 'yes', 'no', True, False - **check_derivatives_for_naninf (str)**: whether to check for NaN / inf in the derivative matrices. Activating this option will cause an error if an invalid number is detected in the constraint Jacobians or the Lagrangian Hessian. If this is not activated, the test is skipped, and the algorithm might proceed with invalid numbers and fail. If test is activated and an invalid number is detected, the matrix is written to output with print_level corresponding to J_MORE_DETAILED; so beware of large output! The default value for this string option is "no". - **jac_c_constant (str or bool)**: Indicates whether to assume that all equality constraints are linear Activating this option will cause Ipopt to ask for the Jacobian of the equality constraints only once from the NLP and reuse this information later. The default value for this string option is "no". Possible values: yes, no, True, False. - **jac_d_constant (str or bool)**: Indicates whether to assume that all inequality constraints are linear Activating this option will cause Ipopt to ask for the Jacobian of the inequality constraints only once from the NLP and reuse this information later. The default value for this string option is "no". Possible values: yes, no, True, False - **hessian_constant (str or bool)**: Indicates whether to assume the problem is a QP (quadratic objective, linear constraints). Activating this option will cause Ipopt to ask for the Hessian of the Lagrangian function only once from the NLP and reuse this information later. The default value for this string option is "no". Possible values: yes, no, True, False. - **nlp_scaling_method (str)**: Select the technique used for scaling the NLP. Selects the technique used for scaling the problem internally before it is solved. For user-scaling, the parameters come from the NLP. If you are using AMPL, they can be specified through suffixes ("scaling_factor") The default value for this string option is "gradient-based". Possible values: - "none": no problem scaling will be performed - "user-scaling": scaling parameters will come from the user - "gradient-based": scale the problem so the maximum gradient at the starting point is ``nlp_scaling_max_gradient`` . - "equilibration-based": scale the problem so that first derivatives are of order 1 at random points (uses Harwell routine MC19) - **obj_scaling_factor** (float): Scaling factor for the objective function. This option sets a scaling factor for the objective function. The scaling is seen internally by Ipopt but the unscaled objective is reported in the console output. If additional scaling parameters are computed (e.g. user-scaling or gradient-based), both factors are multiplied. If this value is chosen to be negative, Ipopt will maximize the objective function instead of minimizing it. The valid range for this real option is unrestricted and its default value is 1. - **nlp_scaling_max_gradient** (float): Maximum gradient after NLP scaling. This is the gradient scaling cut-off. If the maximum gradient is above this value, then gradient based scaling will be performed. Scaling parameters are calculated to scale the maximum gradient back to this value. (This is g_max in Section 3.8 of the implementation paper.) Note: This option is only used if ``nlp_scaling_method`` is chosen as "gradient-based". The valid range for this real option is :math:`0 < \text{nlp_scaling_max_gradient}` and its default value is 100. - **nlp_scaling_obj_target_gradient** (float): advanced! Target value for objective function gradient size. If a positive number is chosen, the scaling factor for the objective function is computed so that the gradient has the max norm of the given size at the starting point. This overrides ``nlp_scaling_max_gradient`` for the objective function. The valid range for this real option is 0 ≤ nlp_scaling_obj_target_gradient and its default value is 0. - **nlp_scaling_constr_target_gradient** (float): arget value for constraint function gradient size. If a positive number is chosen, the scaling factors for the constraint functions are computed so that the gradient has the max norm of the given size at the starting point. This overrides nlp_scaling_max_gradient for the constraint functions. The valid range for this real option is 0 ≤ nlp_scaling_constr_target_gradient and its default value is 0. - **nlp_scaling_min_value** (float): Minimum value of gradient-based scaling values. This is the lower bound for the scaling factors computed by gradient-based scaling method. If some derivatives of some functions are huge, the scaling factors will otherwise become very small, and the (unscaled) final constraint violation, for example, might then be significant. Note: This option is only used if ``nlp_scaling_method`` is chosen as "gradient-based". The valid range for this real option is 0 ≤ nlp_scaling_min_value and its default value is :math:`1e-08`. - **bound_push** (float): Desired minimum absolute distance from the initial point to bound. Determines how much the initial point might have to be modified in order to be sufficiently inside the bounds (together with ``bound_frac`` ). (This is kappa_1 in Section 3.6 of implementation paper.) The valid range for this real option is 0 < bound_push and its default value is 0.01. - **bound_frac** (float): Desired minimum relative distance from the initial point to bound. Determines how much the initial point might have to be modified in order to be sufficiently inside the bounds (together with "bound_push"). (This is kappa_2 in Section 3.6 of implementation paper.) The valid range for this real option is 0 < bound_frac ≤ 0.5 and its default value is 0.01. - **slack_bound_push** (float): Desired minimum absolute distance from the initial slack to bound. Determines how much the initial slack variables might have to be modified in order to be sufficiently inside the inequality bounds (together with ``slack_bound_frac`` ). (This is kappa_1 in Section 3.6 of implementation paper.) The valid range for this real option is 0 < slack_bound_push and its default value is 0.01. - **slack_bound_frac** (float): Desired minimum relative distance from the initial slack to bound. Determines how much the initial slack variables might have to be modified in order to be sufficiently inside the inequality bounds (together with ``slack_bound_push`` ). (This is kappa_2 in Section 3.6 of implementation paper.) The valid range for this real option is 0 < slack_bound_frac ≤ 0.5 and its default value is 0.01. - **constr_mult_init_max** (float): Maximum allowed least-square guess of constraint multipliers. Determines how large the initial least-square guesses of the constraint multipliers are allowed to be (in max-norm). If the guess is larger than this value, it is discarded and all constraint multipliers are set to zero. This options is also used when initializing the restoration phase. By default, "resto.constr_mult_init_max" (the one used in RestoIterateInitializer) is set to zero. The valid range for this real option is 0 ≤ constr_mult_init_max and its default value is 1000. - **bound_mult_init_val** (float): Initial value for the bound multipliers. All dual variables corresponding to bound constraints are initialized to this value. The valid range for this real option is 0 < bound_mult_init_val and its default value is 1. - **bound_mult_init_method (str)**: Initialization method for bound multipliers This option defines how the iterates for the bound multipliers are initialized. If "constant" is chosen, then all bound multipliers are initialized to the value of ``bound_mult_init_val``. If "mu-based" is chosen, the each value is initialized to the the value of "mu_init" divided by the corresponding slack variable. This latter option might be useful if the starting point is close to the optimal solution. The default value for this string option is "constant". Possible values: - "constant": set all bound multipliers to the value of ``bound_mult_init_val`` - "mu-based": initialize to mu_init/x_slack - **least_square_init_primal (str or bool)**: Least square initialization of the primal variables. If set to yes, Ipopt ignores the user provided point and solves a least square problem for the primal variables (x and s) to fit the linearized equality and inequality constraints.This might be useful if the user doesn't know anything about the starting point, or for solving an LP or QP. The default value for this string option is "no". Possible values: - "no": take user-provided point - "yes": overwrite user-provided point with least-square estimates - **least_square_init_duals (str or bool)**: Least square initialization of all dual variables If set to yes, Ipopt tries to compute least-square multipliers (considering ALL dual variables). If successful, the bound multipliers are possibly corrected to be at least ``bound_mult_init_val`` . This might be useful if the user doesn't know anything about the starting point, or for solving an LP or QP. This overwrites option ``bound_mult_init_method`` . The default value for this string option is "no". Possible values: - "no": use ``bound_mult_init_val`` and least-square equality constraint multipliers - "yes": overwrite user-provided point with least-square estimates - **warm_start_init_point (str or bool)**: Warm-start for initial point Indicates whether this optimization should use a warm start initialization, where values of primal and dual variables are given (e.g., from a previous optimization of a related problem.) The default value for this string option is "no". Possible values: - "no" or False: do not use the warm start initialization - "yes" or True: use the warm start initialization - **warm_start_same_structure (str or bool)**: Advanced feature! Indicates whether a problem with a structure identical t the previous one is to be solved. If enabled, then the algorithm assumes that an NLP is now to be solved whose structure is identical to one that already was considered (with the same NLP object). The default value for this string option is "no". Possible values: yes, no, True, False. - **warm_start_bound_push** (float): same as ``bound_push`` for the regular initializer. The valid range for this real option is 0 < warm_start_bound_push and its default value is 0.001. - **warm_start_bound_frac** (float): same as ``bound_frac`` for the regular initializer The valid range for this real option is 0 < warm_start_bound_frac ≤ 0.5 and its default value is 0.001. - **warm_start_slack_bound_push** (float): same as ``slack_bound_push`` for the regular initializer The valid range for this real option is 0 < warm_start_slack_bound_push and its default value is 0.001. - **warm_start_slack_bound_frac** (float): same as ``slack_bound_frac`` for the regular initializer The valid range for this real option is 0 < warm_start_slack_bound_frac ≤ 0.5 and its default value is 0.001. - **warm_start_mult_bound_push** (float): same as ``mult_bound_push`` for the regular initializer The valid range for this real option is 0 < warm_start_mult_bound_push and its default value is 0.001. - **warm_start_mult_init_max** (float): Maximum initial value for the equality multipliers. The valid range for this real option is unrestricted and its default value is :math:`1e+06` . - **warm_start_entire_iterate (str or bool)**: Tells algorithm whether to use the GetWarmStartIterate method in the NLP. The default value for this string option is "no". Possible values: - "no": call GetStartingPoint in the NLP - "yes": call GetWarmStartIterate in the NLP - **warm_start_target_mu** (float): Advanced and experimental! The valid range for this real option is unrestricted and its default value is 0. - **option_file_name (str)**: File name of options file. By default, the name of the Ipopt options file is "ipopt.opt" - or something else if specified in the IpoptApplication::Initialize call. If this option is set by SetStringValue BEFORE the options file is read, it specifies the name of the options file. It does not make any sense to specify this option within the options file. Setting this option to an empty string disables reading of an options file. - **replace_bounds (bool or str)**: Whether all variable bounds should be replaced by inequality constraints. This option must be set for the inexact algorithm. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **skip_finalize_solution_call (str or bool)**: Whether a call to NLP::FinalizeSolution after optimization should be suppressed. In some Ipopt applications, the user might want to call the FinalizeSolution method separately. Setting this option to "yes" will cause the IpoptApplication object to suppress the default call to that method. The default value for this string option is "no". Possible values: "yes", "no", True, False - **timing_statistics (str or bool)**: Indicates whether to measure time spend in components of Ipopt and NLP evaluation. The overall algorithm time is unaffected by this option. The default value for this string option is "no". Possible values: "yes", "no", True, False - **mu_max_fact** (float): Factor for initialization of maximum value for barrier parameter. This option determines the upper bound on the barrier parameter. This upper bound is computed as the average complementarity at the initial point times the value of this option. (Only used if option "mu_strategy" is chosen as "adaptive".) The valid range for this real option is 0 < mu_max_fact and its default value is 1000. - **mu_max** (float): Maximum value for barrier parameter. This option specifies an upper bound on the barrier parameter in the adaptive mu selection mode. If this option is set, it overwrites the effect of mu_max_fact. (Only used if option "mu_strategy" is chosen as "adaptive".) The valid range for this real option is 0 < mu_max and its default value is 100000. - **mu_min** (float): Minimum value for barrier parameter. This option specifies the lower bound on the barrier parameter in the adaptive mu selection mode. By default, it is set to the minimum of :math:`1e-11` and min( ``tol`` , ``compl_inf_tol`` )/( ``barrier_tol_factor`` +1), which should be a reasonable value. (Only used if option ``mu_strategy`` is chosen as "adaptive".) The valid range for this real option is 0 < mu_min and its default value is :math:`1e-11` . - **adaptive_mu_globalization (str)**: Globalization strategy for the adaptive mu selection mode. To achieve global convergence of the adaptive version, the algorithm has to switch to the monotone mode (Fiacco-McCormick approach) when convergence does not seem to appear. This option sets the criterion used to decide when to do this switch. (Only used if option "mu_strategy" is chosen as "adaptive".) The default value for this string option is "obj-constr-filter". Possible values: - "kkt-error": nonmonotone decrease of kkt-error - "obj-constr-filter": 2-dim filter for objective and constraint violation - "never-monotone-mode": disables globalization. - **adaptive_mu_kkterror_red_iters** (float): advanced feature! Maximum number of iterations requiring sufficient progress. For the "kkt-error" based globalization strategy, sufficient progress must be made for "adaptive_mu_kkterror_red_iters" iterations. If this number of iterations is exceeded, the globalization strategy switches to the monotone mode. The valid range for this integer option is 0 ≤ adaptive_mu_kkterror_red_iters and its default value is 4. - **adaptive_mu_kkterror_red_fact** (float): advanced feature! Sufficient decrease factor for "kkt-error" globalization strategy. For the "kkt-error" based globalization strategy, the error must decrease by this factor to be deemed sufficient decrease. The valid range for this real option is 0 < adaptive_mu_kkterror_red_fact < 1 and its default value is 0.9999. - **filter_margin_fact** (float): advanced feature! Factor determining width of margin for obj-constr-filter adaptive globalization strategy. When using the adaptive globalization strategy, "obj-constr-filter", sufficient progress for a filter entry is defined as follows: (new obj) < (filter obj) - filter_margin_fact*(new constr-viol) OR (new constr-viol) < (filter constr-viol) - filter_margin_fact*(new constr-viol). For the description of the "kkt-error-filter" option see ``filter_max_margin`` . The valid range for this real option is 0 < filter_margin_fact < 1 and its default value is :math:`10-05` . - **filter_max_margin** (float): advanced feature! Maximum width of margin in obj-constr-filter adaptive globalization strategy. The valid range for this real option is 0 < filter_max_margin and its default value is 1. - **adaptive_mu_restore_previous_iterate (str or bool)**: advanced feature! Indicates if the previous accepted iterate should be restored if the monotone mode is entered. When the globalization strategy for the adaptive barrier algorithm switches to the monotone mode, it can either start from the most recent iterate (no), or from the last iterate that was accepted (yes). The default value for this string option is "no". Possible values: "yes", "no", True, False - **adaptive_mu_monotone_init_factor** (float): advanced feature! Determines the initial value of the barrier parameter when switching to the monotone mode. When the globalization strategy for the adaptive barrier algorithm switches to the monotone mode and fixed_mu_oracle is chosen as "average_compl", the barrier parameter is set to the current average complementarity times the value of "adaptive_mu_monotone_init_factor". The valid range for this real option is 0 < adaptive_mu_monotone_init_factor and its default value is 0.8. - **adaptive_mu_kkt_norm_type (str)**: advanced! Norm used for the KKT error in the adaptive mu globalization strategies. When computing the KKT error for the globalization strategies, the norm to be used is specified with this option. Note, this option is also used in the QualityFunctionMuOracle. The default value for this string option is "2-norm-squared". Possible values: - "1-norm": use the 1-norm (abs sum) - "2-norm-squared": use the 2-norm squared (sum of squares) - "max-norm": use the infinity norm (max) - "2-norm": use 2-norm - **mu_strategy (str)**: Update strategy for barrier parameter. Determines which barrier parameter update strategy is to be used. The default value for this string option is "monotone". Possible values: - "monotone": use the monotone (Fiacco-McCormick) strategy - "adaptive": use the adaptive update strategy - **mu_oracle (str)**: Oracle for a new barrier parameter in the adaptive strategy. Determines how a new barrier parameter is computed in each "free-mode" iteration of the adaptive barrier parameter strategy. (Only considered if "adaptive" is selected for option "mu_strategy"). The default value for this string option is "quality-function". Possible values: - "probing": Mehrotra's probing heuristic - "loqo": LOQO's centrality rule - "quality-function": minimize a quality function - **fixed_mu_oracle (str)**: Oracle for the barrier parameter when switching to fixed mode. Determines how the first value of the barrier parameter should be computed when switching to the "monotone mode" in the adaptive strategy. (Only considered if "adaptive" is selected for option "mu_strategy".) The default value for this string option is "average_compl". Possible values: - "probing": Mehrotra's probing heuristic - "loqo": LOQO's centrality rule - "quality-function": minimize a quality function - "average_compl": base on current average complementarity - **mu_init** (float): Initial value for the barrier parameter. This option determines the initial value for the barrier parameter (mu). It is only relevant in the monotone, Fiacco-McCormick version of the algorithm. (i.e., if "mu_strategy" is chosen as "monotone") The valid range for this real option is 0 < mu_init and its default value is 0.1. - **barrier_tol_factor** (float): Factor for mu in barrier stop test. The convergence tolerance for each barrier problem in the monotone mode is the value of the barrier parameter times "barrier_tol_factor". This option is also used in the adaptive mu strategy during the monotone mode. This is kappa_epsilon in implementation paper. The valid range for this real option is 0 < barrier_tol_factor and its default value is 10. - **mu_linear_decrease_factor** (float): Determines linear decrease rate of barrier parameter. For the Fiacco-McCormick update procedure the new barrier parameter mu is obtained by taking the minimum of mu*"mu_linear_decrease_factor" and mu^"superlinear_decrease_power". This is kappa_mu in implementation paper. This option is also used in the adaptive mu strategy during the monotone mode. The valid range for this real option is 0 < mu_linear_decrease_factor < 1 and its default value is 0.2. - **mu_superlinear_decrease_power** (float): Determines superlinear decrease rate of barrier parameter. For the Fiacco-McCormick update procedure the new barrier parameter mu is obtained by taking the minimum of mu*"mu_linear_decrease_factor" and mu^"superlinear_decrease_power". This is theta_mu in implementation paper. This option is also used in the adaptive mu strategy during the monotone mode. The valid range for this real option is 1 < mu_superlinear_decrease_power < 2 and its default value is 1.5. - **mu_allow_fast_monotone_decrease (str or bool)**: Advanced feature! Allow skipping of barrier problem if barrier test i already met. The default value for this string option is "yes". Possible values: - "no": Take at least one iteration per barrier problem even if the barrier test is already met for the updated barrier parameter - "yes": Allow fast decrease of mu if barrier test it met - **tau_min** (float): Advanced feature! Lower bound on fraction-to-the-boundary parameter tau. This is tau_min in the implementation paper. This option is also used in the adaptive mu strategy during the monotone mode. The valid range for this real option is 0 < tau_min < 1 and its default value is 0.99. - **sigma_max** (float): Advanced feature! Maximum value of the centering parameter. This is the upper bound for the centering parameter chosen by the quality function based barrier parameter update. Only used if option "mu_oracle" is set to "quality-function". The valid range for this real option is 0 < sigma_max and its default value is 100. - **sigma_min** (float): Advanced feature! Minimum value of the centering parameter. This is the lower bound for the centering parameter chosen by the quality function based barrier parameter update. Only used if option "mu_oracle" is set to "quality-function". The valid range for this real option is 0 ≤ sigma_min and its default value is :math:`10-06` . - **quality_function_norm_type (str)**: Advanced feature. Norm used for components of the quality function. Only used if option "mu_oracle" is set to "quality-function". The default value for this string option is "2-norm-squared". Possible values: - "1-norm": use the 1-norm (abs sum) - "2-norm-squared": use the 2-norm squared (sum of squares) - "max-norm": use the infinity norm (max) - "2-norm": use 2-norm - **quality_function_centrality (str)**: Advanced feature. The penalty term for centrality that is included in quality function. This determines whether a term is added to the quality function to penalize deviation from centrality with respect to complementarity. The complementarity measure here is the xi in the Loqo update rule. Only used if option "mu_oracle" is set to "quality-function". The default value for this string option is "none". Possible values: - "none": no penalty term is added - "log": complementarity * the log of the centrality measure - "reciprocal": complementarity * the reciprocal of the centrality measure - "cubed-reciprocal": complementarity * the reciprocal of the centrality measure cubed - **quality_function_balancing_term (str)**: Advanced feature. The balancing term included in the quality function for centrality. This determines whether a term is added to the quality function that penalizes situations where the complementarity is much smaller than dual and primal infeasibilities. Only used if option "mu_oracle" is set to "quality-function". The default value for this string option is "none". Possible values: - "none": no balancing term is adde - "cubic": :math:`max(0,\max(\text{dual_inf},\text{primal_inf})-\text{compl})^3` - **quality_function_max_section_steps** (int): Maximum number of search steps during direct search procedure determining the optimal centering parameter. The golden section search is performed for the quality function based mu oracle. Only used if option "mu_oracle" is set to "quality-function". The valid range for this integer option is 0 ≤ quality_function_max_section_steps and its default value is 8. - **quality_function_section_sigma_tol** (float): advanced feature! Tolerance for the section search procedure determining the optimal centering parameter (in sigma space). The golden section search is performed for the quality function based mu oracle. Only used if option "mu_oracle" is set to "quality-function". The valid range for this real option is 0 ≤ quality_function_section_sigma_tol < 1 and its default value is 0.01. - **quality_function_section_qf_tol** (float): advanced feature! Tolerance for the golden section search procedure determining the optimal centering parameter (in the function value space). The golden section search is performed for the quality function based mu oracle. Only used if option "mu_oracle" is set to "quality-function". The valid range for this real option is 0 ≤ quality_function_section_qf_tol < 1 and its default value is 0. - **line_search_method (str)**: Advanced feature. Globalization method used in backtracking line search. Only the "filter" choice is officially supported. But sometimes, good results might be obtained with the other choices. The default value for this string option is "filter". Possible values: - "filter": Filter method - "cg-penalty": Chen-Goldfarb penalty function - "penalty": Standard penalty function - **alpha_red_factor** (float): Advanced feature. Fractional reduction of the trial step size in the backtracking lne search. At every step of the backtracking line search, the trial step size is reduced by this factor. The valid range for this real option is 0 < alpha_red_factor < 1 and its default value is 0.5. - **accept_every_trial_step (str or bool)**: Always accept the first trial step. Setting this option to "yes" essentially disables the line search and makes the algorithm take aggressive steps, without global convergence guarantees. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **accept_after_max_steps** (float): advanced feature. Accept a trial point after maximal this number of steps een if it does not satisfy line search conditions. Setting this to -1 disables this option. The valid range for this integer option is -1 ≤ accept_after_max_steps and its default value is -1. - **alpha_for_y (str)**: Method to determine the step size for constraint multipliers (alpha_y) . The default value for this string option is "primal". Possible values: - "primal": use primal step size - "bound-mult": use step size for the bound multipliers (good for LPs) - "min": use the min of primal and bound multipliers - "max": use the max of primal and bound multipliers - "full": take a full step of size one - "min-dual-infeas": choose step size minimizing new dual infeasibility - "safer-min-dual-infeas": like "min_dual_infeas", but safeguarded by "min" and "max" - "primal-and-full": use the primal step size, and full step if delta_x <= alpha_for_y_tol - "dual-and-full": use the dual step size, and full step if delta_x <= alpha_for_y_tol - "acceptor": Call LSAcceptor to get step size for y - **alpha_for_y_tol** (float): Tolerance for switching to full equality multiplier steps. This is only relevant if "alpha_for_y" is chosen "primal-and-full" or "dual-and-full". The step size for the equality constraint multipliers is taken to be one if the max-norm of the primal step is less than this tolerance. The valid range for this real option is 0 ≤ alpha_for_y_tol and its default value is 10. - **tiny_step_tol** (float): Advanced feature. Tolerance for detecting numerically insignificant steps. If the search direction in the primal variables (x and s) is, in relative terms for each component, less than this value, the algorithm accepts the full step without line search. If this happens repeatedly, the algorithm will terminate with a corresponding exit message. The default value is 10 times machine precision. The valid range for this real option is 0 ≤ tiny_step_tol and its default value is 2.22045 · :math:`1e-15`. - **tiny_step_y_tol** (float): Advanced feature. Tolerance for quitting because of numerically insignificant steps. If the search direction in the primal variables (x and s) is, in relative terms for each component, repeatedly less than tiny_step_tol, and the step in the y variables is smaller than this threshold, the algorithm will terminate. The valid range for this real option is 0 ≤ tiny_step_y_tol and its default value is 0.01. - **watchdog_shortened_iter_trigger** (int): Number of shortened iterations that trigger the watchdog. If the number of successive iterations in which the backtracking line search did not accept the first trial point exceeds this number, the watchdog procedure is activated. Choosing "0" here disables the watchdog procedure. The valid range for this integer option is 0 ≤ watchdog_shortened_iter_trigger and its default value is 10. - **watchdog_trial_iter_max** (int): Maximum number of watchdog iterations. This option determines the number of trial iterations allowed before the watchdog procedure is aborted and the algorithm returns to the stored point. The valid range for this integer option is 1 ≤ watchdog_trial_iter_max and its default value is 3. theta_max_fact (float): Advanced feature. Determines upper bound for constraint violation in the filter. The algorithmic parameter theta_max is determined as theta_max_fact times the maximum of 1 and the constraint violation at initial point. Any point with a constraint violation larger than theta_max is unacceptable to the filter (see Eqn. (21) in the implementation paper). The valid range for this real option is 0 < theta_max_fact and its default value is 10000. - **theta_min_fact** (float): advanced feature. Determines constraint violation threshold in the switching rule. The algorithmic parameter theta_min is determined as theta_min_fact times the maximum of 1 and the constraint violation at initial point. The switching rules treats an iteration as an h-type iteration whenever the current constraint violation is larger than theta_min (see paragraph before Eqn. (19) in the implementation paper). The valid range for this real option is 0 < theta_min_fact and its default value is 0.0001. - **eta_phi** (float): advanced! Relaxation factor in the Armijo condition. See Eqn. (20) in the implementation paper. The valid range for this real option is 0 < eta_phi < 0.5 and its default value is :math:`1e-08`. - **delta** (float): advanced! Multiplier for constraint violation in the switching rule. See Eqn. (19) in the implementation paper. The valid range for this real option is 0 < delta and its default value is 1. - **s_phi** (float): advanced! Exponent for linear barrier function model in the switching rule. See Eqn. (19) in the implementation paper. The valid range for this real option is 1 < s_phi and its default value is 2.3. - **s_theta** (float): advanced! Exponent for current constraint violation in the switching rule. See Eqn. (19) in the implementation paper. The valid range for this real option is 1 < s_theta and its default value is 1.1. - **gamma_phi** (float): advanced! Relaxation factor in the filter margin for the barrier function. See Eqn. (18a) in the implementation paper. The valid range for this real option is 0 < gamma_phi < 1 and its default value is :math:`1e-08`. - **gamma_theta** (float): advanced! Relaxation factor in the filter margin for the constraint violation. See Eqn. (18b) in the implementation paper. The valid range for this real option is 0 < gamma_theta < 1 and its default value is :math:`1e-05`. - **alpha_min_frac** (float): advanced! Safety factor for the minimal step size (before switching to restoration phase). This is gamma_alpha in Eqn. (20) in the implementation paper. The valid range for this real option is 0 < alpha_min_frac < 1 and its default value is 0.05. - **max_soc** (int): Maximum number of second order correction trial steps at each iteration. Choosing 0 disables the second order corrections. This is p^{max} of Step A-5.9 of Algorithm A in the implementation paper. The valid range for this integer option is 0 ≤ max_soc and its default value is 4. - **kappa_soc** (float): advanced! Factor in the sufficient reduction rule for second order correction. This option determines how much a second order correction step must reduce the constraint violation so that further correction steps are attempted. See Step A-5.9 of Algorithm A in the implementation paper. The valid range for this real option is 0 < kappa_soc and its default value is 0.99. - **obj_max_inc** (float): advanced! Determines the upper bound on the acceptable increase of barrier objective function. Trial points are rejected if they lead to an increase in the barrier objective function by more than obj_max_inc orders of magnitude. The valid range for this real option is 1 < obj_max_inc and its default value is 5. - **max_filter_resets** (int): advanced! Maximal allowed number of filter resets. A positive number enables a heuristic that resets the filter, whenever in more than "filter_reset_trigger" successive iterations the last rejected trial steps size was rejected because of the filter. This option determine the maximal number of resets that are allowed to take place. The valid range for this integer option is 0 ≤ max_filter_resets and its default value is 5. - **filter_reset_trigger** (int): Advanced! Number of iterations that trigger the filter reset. If the filter reset heuristic is active and the number of successive iterations in which the last rejected trial step size was rejected because of the filter, the filter is reset. The valid range for this integer option is 1 ≤ filter_reset_trigger and its default value is 5. - **corrector_type (str)**: advanced! The type of corrector steps that should be taken. If "mu_strategy" is "adaptive", this option determines what kind of corrector steps should be tried. Changing this option is experimental. The default value for this string option is "none". Possible values: - "none" or None: no corrector - "affine": corrector step towards mu=0 - "primal-dual": corrector step towards current mu - **skip_corr_if_neg_curv (str or bool)**: advanced! Whether to skip the corrector step in negative curvature iteration. The corrector step is not tried if negative curvature has been encountered during the computation of the search direction in the current iteration. This option is only used if "mu_strategy" is "adaptive". Changing this option is experimental. The default value for this string option is "yes". Possible values: "yes", "no", True, False. - **skip_corr_in_monotone_mode (str or bool)**: Advanced! Whether to skip the corrector step during monotone brrier parameter mode. The corrector step is not tried if the algorithm is currently in the monotone mode (see also option "barrier_strategy"). This option is only used if "mu_strategy" is "adaptive". Changing this option is experimental. The default value for this string option is "yes". Possible values: "yes", "no", True, False - **corrector_compl_avrg_red_fact** (float): advanced! Complementarity tolerance factor for accepting corrector step. This option determines the factor by which complementarity is allowed to increase for a corrector step to be accepted. Changing this option is experimental. The valid range for this real option is 0 < corrector_compl_avrg_red_fact and its default value is 1. - **soc_method** (int): Ways to apply second order correction. This option determines the way to apply second order correction, 0 is the method described in the implementation paper. 1 is the modified way which adds alpha on the rhs of x and s rows. Officially, the valid range for this integer option is 0 ≤ soc_method ≤ 1 and its default value is 0 but only 0 and 1 are allowed. - **nu_init** (float): advanced! Initial value of the penalty parameter. The valid range for this real option is 0 < nu_init and its default value is :math:`1e-06`. - **nu_inc** (float): advanced! Increment of the penalty parameter. The valid range for this real option is 0 < nu_inc and its default value is 0.0001. - **rho** (float): advanced! Value in penalty parameter update formula. The valid range for this real option is 0 < rho < 1 and its default value is 0.1. - **kappa_sigma** (float): advanced! Factor limiting the deviation of dual variables from primal estimates. If the dual variables deviate from their primal estimates, a correction is performed. See Eqn. (16) in the implementation paper. Setting the value to less than 1 disables the correction. The valid range for this real option is 0 < kappa_sigma and its default value is :math:`1e+10`. - **recalc_y (str or bool)**: Tells the algorithm to recalculate the equality and inequality multipliers as least square estimates. This asks the algorithm to recompute the multipliers, whenever the current infeasibility is less than recalc_y_feas_tol. Choosing yes might be helpful in the quasi-Newton option. However, each recalculation requires an extra factorization of the linear system. If a limited memory quasi-Newton option is chosen, this is used by default. The default value for this string option is "no". Possible values: - "no" or False: use the Newton step to update the multipliers - "yes" or True: use least-square multiplier - **estimates recalc_y_feas_tol** (float): Feasibility threshold for recomputation of multipliers. If recalc_y is chosen and the current infeasibility is less than this value, then the multipliers are recomputed. The valid range for this real option is 0 < recalc_y_feas_tol and its default value is :math:`1e-06`. - **slack_move** (float): advanced! Correction size for very small slacks. Due to numerical issues or the lack of an interior, the slack variables might become very small. If a slack becomes very small compared to machine precision, the corresponding bound is moved slightly. This parameter determines how large the move should be. Its default value is mach_eps^{3/4}. See also end of Section 3.5 in implementation paper - but actual implementation might be somewhat different. The valid range for this real option is 0 ≤ slack_move and its default value is 1.81899 · :math:`1e-12`. - **constraint_violation_norm_type (str)**: advanced! Norm to be used for the constraint violation in te line search. Determines which norm should be used when the algorithm computes the constraint violation in the line search. The default value for this string option is "1-norm". Possible values: - "1-norm": use the 1-norm - "2-norm": use the 2-norm - "max-norm": use the infinity norm - **mehrotra_algorithm (str or bool)**: Indicates whether to do Mehrotra's predictor-corrector algorithm. If enabled, line search is disabled and the (unglobalized) adaptive mu strategy is chosen with the "probing" oracle, and "corrector_type=affine" is used without any safeguards; you should not set any of those options explicitly in addition. Also, unless otherwise specified, the values of ``bound_push`` , ``bound_frac`` , and ``bound_mult_init_val`` are set more aggressive, and sets "alpha_for_y=bound_mult". The Mehrotra's predictor-corrector algorithm works usually very well for LPs and convex QPs. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **fast_step_computation (str or bool)**: Indicates if the linear system should be solved quickly. If enabled, the algorithm assumes that the linear system that is solved to obtain the search direction is solved sufficiently well. In that case, no residuals are computed to verify the solution and the computation of the search direction is a little faster. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **min_refinement_steps** (int): Minimum number of iterative refinement steps per linear system solve. Iterative refinement (on the full asymmetric system) is performed for each right hand side. This option determines the minimum number of iterative refinements (i.e. at least "min_refinement_steps" iterative refinement steps are enforced per right hand side.) The valid range for this integer option is 0 ≤ min_refinement_steps and its default value is 1. - **max_refinement_steps** (int): Maximum number of iterative refinement steps per linear system solve. Iterative refinement (on the full unsymmetric system) is performed for each right hand side. This option determines the maximum number of iterative refinement steps. The valid range for this integer option is 0 ≤ max_refinement_steps and its default value is 10. - **residual_ratio_max** (float): advanced! Iterative refinement tolerance. Iterative refinement is performed until the residual test ratio is less than this tolerance (or until "max_refinement_steps" refinement steps are performed). The valid range for this real option is 0 < residual_ratio_max and its default value is :math:`1e-10`. - **residual_ratio_singular** (float): advanced! Threshold for declaring linear system singular after filed iterative refinement. If the residual test ratio is larger than this value after failed iterative refinement, the algorithm pretends that the linear system is singular. The valid range for this real option is 0 < residual_ratio_singular and its default value is :math:`1e-05`. - **residual_improvement_factor** (float): advanced! Minimal required reduction of residual test ratio in iterative refinement. If the improvement of the residual test ratio made by one iterative refinement step is not better than this factor, iterative refinement is aborted. The valid range for this real option is 0 < residual_improvement_factor and its default value is 1. - **neg_curv_test_tol** (float): Tolerance for heuristic to ignore wrong inertia. If nonzero, incorrect inertia in the augmented system is ignored, and Ipopt tests if the direction is a direction of positive curvature. This tolerance is alpha_n in the paper by :cite:`Chiang2014` and it determines when the direction is considered to be sufficiently positive. A value in the range of [1e-12, 1e-11] is recommended. The valid range for this real option is 0 ≤ neg_curv_test_tol and its default value is 0. - **neg_curv_test_reg (str or bool)**: Whether to do the curvature test with the primal regularization (see :cite:`Chiang2014`). The default value for this string option is "yes". Possible values: - "yes" or True: use primal regularization with the inertia-free curvature test - "no" or False: use original IPOPT approach, in which the primal regularization is ignored - **max_hessian_perturbation** (float): Maximum value of regularization parameter for handling negative curvature. In order to guarantee that the search directions are indeed proper descent directions, Ipopt requires that the inertia of the (augmented) linear system for the step computation has the correct number of negative and positive eigenvalues. The idea is that this guides the algorithm away from maximizers and makes Ipopt more likely converge to first order optimal points that are minimizers. If the inertia is not correct, a multiple of the identity matrix is added to the Hessian of the Lagrangian in the augmented system. This parameter gives the maximum value of the regularization parameter. If a regularization of that size is not enough, the algorithm skips this iteration and goes to the restoration phase. This is delta_w^max in the implementation paper. The valid range for this real option is 0 < max_hessian_perturbation and its default value is :math:`1e+20`. - **min_hessian_perturbation** (float): Smallest perturbation of the Hessian block. The size of the perturbation of the Hessian block is never selected smaller than this value, unless no perturbation is necessary. This is delta_w^min in implementation paper. The valid range for this real option is 0 ≤ min_hessian_perturbation and its default value is :math:`1e-20`. - **perturb_inc_fact_first** (float): Increase factor for x-s perturbation for very first perturbation. The factor by which the perturbation is increased when a trial value was not sufficient - this value is used for the computation of the very first perturbation and allows a different value for the first perturbation than that used for the remaining perturbations. This is bar_kappa_w^+ in the implementation paper. The valid range for this real option is 1 < perturb_inc_fact_first and its default value is 100. - **perturb_inc_fact** (float): Increase factor for x-s perturbation. The factor by which the perturbation is increased when a trial value was not sufficient - this value is used for the computation of all perturbations except for the first. This is kappa_w^+ in the implementation paper. The valid range for this real option is 1 < perturb_inc_fact and its default value is 8. - **perturb_dec_fact** (float): Decrease factor for x-s perturbation. The factor by which the perturbation is decreased when a trial value is deduced from the size of the most recent successful perturbation. This is kappa_w^- in the implementation paper. The valid range for this real option is 0 < perturb_dec_fact < 1 and its default value is 0.333333. - **first_hessian_perturbation** (float): Size of first x-s perturbation tried. The first value tried for the x-s perturbation in the inertia correction scheme. This is delta_0 in the implementation paper. The valid range for this real option is 0 < first_hessian_perturbation and its default value is 0.0001. - **jacobian_regularization_value** (float): Size of the regularization for rank-deficient constraint Jacobians. This is bar delta_c in the implementation paper. The valid range for this real option is 0 ≤ jacobian_regularization_value and its default value is :math:`1e-08`. - **jacobian_regularization_exponent** (float): advanced! Exponent for mu in the regularization for rnk-deficient constraint Jacobians. This is kappa_c in the implementation paper. The valid range for this real option is 0 ≤ jacobian_regularization_exponent and its default value is 0.25. - **perturb_always_cd (str or bool)**: advanced! Active permanent perturbation of constraint linearization. Enabling this option leads to using the delta_c and delta_d perturbation for the computation of every search direction. Usually, it is only used when the iteration matrix is singular. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **expect_infeasible_problem (str or bool)**: Enable heuristics to quickly detect an infeasible problem. This options is meant to activate heuristics that may speed up the infeasibility determination if you expect that there is a good chance for the problem to be infeasible. In the filter line search procedure, the restoration phase is called more quickly than usually, and more reduction in the constraint violation is enforced before the restoration phase is left. If the problem is square, this option is enabled automatically. The default value for this string option is "no". Possible values: "yes", "no", True, False. - **expect_infeasible_problem_ctol** (float): Threshold for disabling "expect_infeasible_problem" option. If the constraint violation becomes smaller than this threshold, the "expect_infeasible_problem" heuristics in the filter line search are disabled. If the problem is square, this options is set to 0. The valid range for this real option is 0 ≤ expect_infeasible_problem_ctol and its default value is 0.001. - **expect_infeasible_problem_ytol** (float): Multiplier threshold for activating "xpect_infeasible_problem" option. If the max norm of the constraint multipliers becomes larger than this value and "expect_infeasible_problem" is chosen, then the restoration phase is entered. The valid range for this real option is 0 < expect_infeasible_problem_ytol and its default value is :math:`1e+08`. - **start_with_resto (str or bool)**: Whether to switch to restoration phase in first iteration.Setting this option to "yes" forces the algorithm to switch to the feasibility restoration phase in the first iteration. If the initial point is feasible, the algorithm will abort with a failure. The default value for this string option is "no". Possible values: "yes", "no", True, False - **soft_resto_pderror_reduction_factor** (float): Required reduction in primal-dual error in the soft restoration phase. The soft restoration phase attempts to reduce the primal-dual error with regular steps. If the damped primal-dual step (damped only to satisfy the fraction-to-the-boundary rule) is not decreasing the primal-dual error by at least this factor, then the regular restoration phase is called. Choosing "0" here disables the soft restoration phase. The valid range for this real option is 0 ≤ soft_resto_pderror_reduction_factor and its default value is 0.9999. - **max_soft_resto_iters** (int): advanced! Maximum number of iterations performed successively in soft rstoration phase. If the soft restoration phase is performed for more than so many iterations in a row, the regular restoration phase is called. The valid range for this integer option is 0 ≤ max_soft_resto_iters and its default value is 10. - **required_infeasibility_reduction** (float): Required reduction of infeasibility before leaving restoration phase. The restoration phase algorithm is performed, until a point is found that is acceptable to the filter and the infeasibility has been reduced by at least the fraction given by this option. The valid range for this real option is 0 ≤ required_infeasibility_reduction < 1 and its default value is 0.9. - **max_resto_iter** (int): advanced! Maximum number of successive iterations in restoration phase.The algorithm terminates with an error message if the number of iterations successively taken in the restoration phase exceeds this number. The valid range for this integer option is 0 ≤ max_resto_iter and its default value is 3000000. - **evaluate_orig_obj_at_resto_trial (str or bool)**: Determines if the original objective function should be evaluated at restoration phase trial points. Enabling this option makes the restoration phase algorithm evaluate the objective function of the original problem at every trial point encountered during the restoration phase, even if this value is not required. In this way, it is guaranteed that the original objective function can be evaluated without error at all accepted iterates; otherwise the algorithm might fail at a point where the restoration phase accepts an iterate that is good for the restoration phase problem, but not the original problem. On the other hand, if the evaluation of the original objective is expensive, this might be costly. The default value for this string option is "yes". Possible values: "yes", "no", True, False - **resto_penalty_parameter** (float): advanced! Penalty parameter in the restoration phase objective function. This is the parameter rho in equation (31a) in the Ipopt implementation paper. The valid range for this real option is 0 < resto_penalty_parameter and its default value is 1000. - **resto_proximity_weight** (float): advanced! Weighting factor for the proximity term in restoration pase objective. This determines how the parameter zeta in equation (29a) in the implementation paper is computed. zeta here is resto_proximity_weight*sqrt(mu), where mu is the current barrier parameter. The valid range for this real option is 0 ≤ resto_proximity_weight and its default value is 1. - **bound_mult_reset_threshold** (float): Threshold for resetting bound multipliers after the restoration pase. After returning from the restoration phase, the bound multipliers are updated with a Newton step for complementarity. Here, the change in the primal variables during the entire restoration phase is taken to be the corresponding primal Newton step. However, if after the update the largest bound multiplier exceeds the threshold specified by this option, the multipliers are all reset to 1. The valid range for this real option is 0 ≤ bound_mult_reset_threshold and its default value is 1000. - **constr_mult_reset_threshold** (float): Threshold for resetting equality and inequality multipliers ater restoration phase. After returning from the restoration phase, the constraint multipliers are recomputed by a least square estimate. This option triggers when those least-square estimates should be ignored. The valid range for this real option is 0 ≤ constr_mult_reset_threshold and its default value is 0. - **resto_failure_feasibility_threshold** (float): advanced! Threshold for primal infeasibility to declare failure of restoration phase. If the restoration phase is terminated because of the "acceptable" termination criteria and the primal infeasibility is smaller than this value, the restoration phase is declared to have failed. The default value is actually 1e2*tol, where tol is the general termination tolerance. The valid range for this real option is 0 ≤ resto_failure_feasibility_threshold and its default value is 0. - **limited_memory_aug_solver (str)**: advanced! Strategy for solving the augmented system for low-rank Hessian. The default value for this string option is "sherman-morrison". Possible values: - "sherman-morrison": use Sherman-Morrison formula - "extended": use an extended augmented system - **limited_memory_max_history** (int): Maximum size of the history for the limited quasi-Newton Hessian approximation. This option determines the number of most recent iterations that are taken into account for the limited-memory quasi-Newton approximation. The valid range for this integer option is 0 ≤ limited_memory_max_history and its default value is 6. - **limited_memory_update_type (str)**: Quasi-Newton update formula for the limited memory quasi-Newton approximation. The default value for this string option is "bfgs". Possible values: - "bfgs": BFGS update (with skipping) - "sr1": SR1 (not working well) - **limited_memory_initialization (str)**: Initialization strategy for the limited memory quasi-Newton aproximation. Determines how the diagonal Matrix B_0 as the first term in the limited memory approximation should be computed. The default value for this string option is "scalar1". Possible values: - "scalar1": sigma = s^Ty/s^Ts - "scalar2": sigma = y^Ty/s^Ty - "scalar3": arithmetic average of scalar1 and scalar2 - "scalar4": geometric average of scalar1 and scalar2 - "constant": sigma = limited_memory_init_val - **limited_memory_init_val** (float): Value for B0 in low-rank update. The starting matrix in the low rank update, B0, is chosen to be this multiple of the identity in the first iteration (when no updates have been performed yet), and is constantly chosen as this value, if "limited_memory_initialization" is "constant". The valid range for this real option is 0 < limited_memory_init_val and its default value is 1. - **limited_memory_init_val_max** (float): Upper bound on value for B0 in low-rank update. The starting matrix in the low rank update, B0, is chosen to be this multiple of the identity in the first iteration (when no updates have been performed yet), and is constantly chosen as this value, if "limited_memory_initialization" is "constant". The valid range for this real option is 0 < limited_memory_init_val_max and its default value is :math:`1e+08`. - **limited_memory_init_val_min** (float): Lower bound on value for B0 in low-rank update. The starting matrix in the low rank update, B0, is chosen to be this multiple of the identity in the first iteration (when no updates have been performed yet), and is constantly chosen as this value, if "limited_memory_initialization" is "constant". The valid range for this real option is 0 < limited_memory_init_val_min and its default value is :math:`1e-08`. - **limited_memory_max_skipping** (int): Threshold for successive iterations where update is skipped. If the update is skipped more than this number of successive iterations, the quasi-Newton approximation is reset. The valid range for this integer option is 1 ≤ limited_memory_max_skipping and its default value is 2. - **limited_memory_special_for_resto (str or bool)**: Determines if the quasi-Newton updates should be special dring the restoration phase. Until Nov 2010, Ipopt used a special update during the restoration phase, but it turned out that this does not work well. The new default uses the regular update procedure and it improves results. If for some reason you want to get back to the original update, set this option to "yes". The default value for this string option is "no". Possible values: "yes", "no", True, False. - **hessian_approximation (str)**: Indicates what Hessian information is to be used. This determines which kind of information for the Hessian of the Lagrangian function is used by the algorithm. The default value for this string option is "limited-memory". Possible values: - "exact": Use second derivatives provided by the NLP. - "limited-memory": Perform a limited-memory quasi-Newton approximation - **hessian_approximation_space (str)**: advanced! Indicates in which subspace the Hessian information is to be approximated. The default value for this string option is "nonlinear-variables". Possible values: - "nonlinear-variables": only in space of nonlinear variables. - "all-variables": in space of all variables (without slacks) - **linear_solver (str)**: Linear solver used for step computations. Determines which linear algebra package is to be used for the solution of the augmented linear system (for obtaining the search directions). The default value for this string option is "ma27". Possible values: - "mumps" (use the Mumps package, default) - "ma27" (load the Harwell routine MA27 from library at runtime) - "ma57" (load the Harwell routine MA57 from library at runtime) - "ma77" (load the Harwell routine HSL_MA77 from library at runtime) - "ma86" (load the Harwell routine MA86 from library at runtime) - "ma97" (load the Harwell routine MA97 from library at runtime) - "pardiso" (load the Pardiso package from pardiso-project.org from user-provided library at runtime) - "custom" (use custom linear solver (expert use)) - **linear_solver_options** (dict or None): dictionary with the linear solver options, possibly including `linear_system_scaling`, `hsllib` and `pardisolib`. See the `ipopt documentation for details `_. The linear solver options are not automatically converted to float at the moment.] ``` (fides-algorithm)= ## The Fides Optimizer optimagic supports the [Fides Optimizer](https://fides-optimizer.readthedocs.io/en/latest). To use Fides, you need to have [the fides package](https://github.com/fides-dev/fides) installed (`pip install fides>=0.7.4`, make sure you have at least 0.7.1). ```{eval-rst} .. dropdown:: fides .. code-block:: "fides" `Fides `_ implements an Interior Trust Region Reflective for boundary costrained optimization problems based on the papers :cite:`Coleman1994` and :cite:`Coleman1996`. Accordingly, Fides is named after the Roman goddess of trust and reliability. In contrast to other optimizers, Fides solves the full trust-region subproblem exactly, which can yields higher quality proposal steps, but is computationally more expensive. This makes Fides particularly attractive for optimization problems with objective functions that are computationally expensive to evaluate and the computational cost of solving the trust-region subproblem is negligible. - **hessian_update_strategy** (str): Hessian Update Strategy to employ. You can provide a lowercase or uppercase string or a fides.hession_approximation.HessianApproximation class instance. FX, SSM, TSSM and GNSBFGS are not supported by optimagic. The available update strategies are: - **bb**: Broydens "bad" method as introduced :cite:`Broyden1965`. - **bfgs**: Broyden-Fletcher-Goldfarb-Shanno update strategy. - **bg**: Broydens "good" method as introduced in :cite:`Broyden1965`. - You can use a general BroydenClass Update scheme using the Broyden class from `fides.hessian_approximation`. This is a generalization of BFGS/DFP methods where the parameter :math:`phi` controls the convex combination between the two. This is a rank 2 update strategy that preserves positive-semidefiniteness and symmetry (if :math:`\phi \in [0,1]`). It is described in :cite:`Nocedal1999`, Chapter 6.3. - **dfp**: Davidon-Fletcher-Powell update strategy. - **sr1**: Symmetric Rank 1 update strategy as described in :cite:`Nocedal1999`, Chapter 6.2. - **convergence.ftol_abs** (float): absolute convergence criterion tolerance. This is only the interpretation of this parameter if the relative criterion tolerance is set to 0. Denoting the absolute criterion tolerance by :math:`\alpha` and the relative criterion tolerance by :math:`\beta`, the convergence condition on the criterion improvement is :math:`|f(x_k) - f(x_{k-1})| < \alpha + \beta \cdot |f(x_{k-1})|` - **convergence.ftol_rel** (float): relative convergence criterion tolerance. This is only the interpretation of this parameter if the absolute criterion tolerance is set to 0 (as is the default). Denoting the absolute criterion tolerance by :math:`\alpha` and the relative criterion tolerance by :math:`\beta`, the convergence condition on the criterion improvement is :math:`|f(x_k) - f(x_{k-1})| < \alpha + \beta \cdot |f(x_{k-1})|` - **convergence.xtol_abs** (float): The optimization terminates successfully when the step size falls below this number, i.e. when :math:`||x_{k+1} - x_k||` is smaller than this tolerance. - **convergence.gtol_abs** (float): The optimization terminates successfully when the gradient norm is less or equal than this tolerance. - **convergence.gtol_rel** (float): The optimization terminates successfully when the norm of the gradient divided by the absolute function value is less or equal to this tolerance. - **stopping.maxiter** (int): maximum number of allowed iterations. - **stopping.max_seconds** (int): maximum number of walltime seconds, deactivated by default. - **trustregion.initial_radius** (float): Initial trust region radius. Default is 1. - **trustregion.stepback_strategy** (str): search refinement strategy if proposed step reaches a parameter bound. The default is "truncate". The available options are: - "reflect": recursive reflections at boundary. - "reflect_single": single reflection at boundary. - "truncate": truncate step at boundary and re-solve the restricted subproblem - "mixed": mix reflections and truncations - **trustregion.subspace_dimension** (str): Subspace dimension in which the subproblem will be solved. The default is "2D". The following values are available: - "2D": Two dimensional Newton/Gradient subspace - "full": full dimensionality - "scg": Conjugated Gradient subspace via Steihaug's method - **trustregion.max_stepback_fraction** (float): Stepback parameter that controls how close steps are allowed to get to the boundary. It is the maximal fraction of a step to take if full step would reach breakpoint. - **trustregion.decrease_threshold** (float): Acceptance threshold for trust region ratio. The default is 0.25 (:cite:`Nocedal2006`). The radius is decreased if the trust region ratio is below this value. This is denoted by :math:`\\mu` in algorithm 4.1 in :cite:`Nocedal2006`. - **trustregion.increase_threshold** (float): Threshold for the trust region radius ratio above which the trust region radius can be increased. This is denoted by :math:`\eta` in algorithm 4.1 in :cite:`Nocedal2006`. The default is 0.75 (:cite:`Nocedal2006`). - **trustregion.decrease_factor** (float): factor by which trust region radius will be decreased in case it is decreased. This is denoted by :math:`\gamma_1` in algorithm 4.1 in :cite:`Nocedal2006` and its default is 0.25. - **trustregion.increase_factor** (float): factor by which trust region radius will be increase in case it is increase. This is denoted by :math:`\gamma_2` in algorithm 4.1 in :cite:`Nocedal2006` and its default is 2.0. - **trustregion.refine_stepback** (bool): whether to refine stepbacks via optimization. Default is False. - **trustregion.scaled_gradient_as_possible_stepback** (bool): whether the scaled gradient should be added to the set of possible stepback proposals. Default is False. ``` ## The NLOPT Optimizers (nlopt) optimagic supports the following [NLOPT](https://nlopt.readthedocs.io/en/latest/) algorithms. Please add the [appropriate citations](https://nlopt.readthedocs.io/en/latest/Citing_NLopt/) in addition to optimagic when using an NLOPT algorithm. To install nlopt run `conda install nlopt`. ```{eval-rst} .. dropdown:: nlopt_bobyqa .. code-block:: "nlopt_bobyqa" Minimize a scalar function using the BOBYQA algorithm. The implementation is derived from the BOBYQA subroutine of M. J. D. Powell. The algorithm performs derivative free bound-constrained optimization using an iteratively constructed quadratic approximation for the objective function. Due to its use of quadratic appoximation, the algorithm may perform poorly for objective functions that are not twice-differentiable. For details see :cite:`Powell2009`. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_neldermead .. code-block:: "nlopt_neldermead" Minimize a scalar function using the Nelder-Mead simplex algorithm. The basic algorithm is described in :cite:`Nelder1965`. The difference between the nlopt implementation an the original implementation is that the nlopt version supports bounds. This is done by moving all new points that would lie outside the bounds exactly on the bounds. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_praxis .. code-block:: "nlopt_praxis" Minimize a scalar function using principal-axis method. This is a gradient-free local optimizer originally described in :cite:`Brent1972`. It assumes quadratic form of the optimized function and repeatedly updates a set of conjugate search directions. The algorithm is not invariant to scaling of the objective function and may fail under its certain rank-preserving transformations (e.g., will lead to a non-quadratic shape of the objective function). The algorithm is not determenistic and it is not possible to achieve detereminancy via seed setting. The algorithm failed on a simple benchmark function with finite parameter bounds. Passing arguments `lower_bounds` and `upper_bounds` has been disabled for this algorithm. The difference between the nlopt implementation an the original implementation is that the nlopt version supports bounds. This is done by returning infinity (Inf) when the constraints are violated. The implementation of bound constraints is achieved at the const of significantly reduced speed of convergence. In case of bounded constraints, this method is dominated by `nlopt_bobyqa` and `nlopt_cobyla`. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_cobyla .. code-block:: "nlopt_cobyla" Minimize a scalar function using the cobyla method. The alggorithm is derived from Powell's Constrained Optimization BY Linear Approximations (COBYLA) algorithm. It is a derivative-free optimizer with nonlinear inequality and equality constrains, described in :cite`Powell1994`. It constructs successive linear approximations of the objective function and constraints via a simplex of n+1 points (in n dimensions), and optimizes these approximations in a trust region at each step. The the nlopt implementation differs from the original implementation in a a few ways: - Incorporates all of the NLopt termination criteria. - Adds explicit support for bound constraints. - Allows the algorithm to increase the trust-reion radius if the predicted imptoovement was approximately right and the simplex is satisfactory. - Pseudo-randomizes simplex steps in the algorithm, aimproving robustness by avoiding accidentally taking steps that don't improve conditioning, preserving the deterministic nature of the algorithm. - Supports unequal initial-step sizes in the different parameters. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_sbplx .. code-block:: "nlopt_sbplx" Minimize a scalar function using the "Subplex" algorithm. The alggorithm is a reimplementation of Tom Rowan's "Subplex" algorithm. See :cite:`Rowan1990`. Subplex is a variant of Nedler-Mead that uses Nedler-Mead on a sequence of subspaces. It is climed to be more efficient and robust than the original Nedler-Mead algorithm. The difference between this re-implementation and the original algorithm of Rowan, is that it explicitly supports bound constraints providing big improvement in the case where the optimum lies against one of the constraints. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_newuoa .. code-block:: "nlopt_newuoa" Minimize a scalar function using the NEWUOA algorithm. The algorithm is derived from the NEWUOA subroutine of M.J.D Powell which uses iteratively constructed quadratic approximation of the objctive fucntion to perform derivative-free unconstrained optimization. Fore more details see: :cite:`Powell2004`. The algorithm in `nlopt` has been modified to support bound constraints. If all of the bound constraints are infinite, this function calls the `nlopt.LN_NEWUOA` optimizers for uncsonstrained optimization. Otherwise, the `nlopt.LN_NEWUOA_BOUND` optimizer for constrained problems. `NEWUOA` requires the dimension n of the parameter space to be `≥ 2`, i.e. the implementation does not handle one-dimensional optimization problems. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_tnewton .. code-block:: "nlopt_tnewton" Minimize a scalar function using the "TNEWTON" algorithm. The alggorithm is based on a Fortran implementation of a preconditioned inexact truncated Newton algorithm written by Prof. Ladislav Luksan. Truncated Newton methods are a set of algorithms designed to solve large scale optimization problems. The algorithms use (inaccurate) approximations of the solutions to Newton equations, using conjugate gradient methodds, to handle the expensive calculations of derivatives during each iteration. Detailed description of algorithms is given in :cite:`Dembo1983`. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_lbfgs .. code-block:: "nlopt_lbfgs" Minimize a scalar function using the "LBFGS" algorithm. The alggorithm is based on a Fortran implementation of low storage BFGS algorithm written by Prof. Ladislav Luksan. LFBGS is an approximation of the original Broyden–Fletcher–Goldfarb–Shanno algorithm based on limited use of memory. Memory efficiency is obtained by preserving a limi- ted number (<10) of past updates of candidate points and gradient values and using them to approximate the hessian matrix. Detailed description of algorithms is given in :cite:`Nocedal1989`, :cite:`Nocedal1980`. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_ccsaq .. code-block:: "nlopt_ccsaq" Minimize a scalar function using CCSAQ algorithm. CCSAQ uses the quadratic variant of the conservative convex separable approximation. The algorithm performs gradient based local optimization with equality (but not inequality) constraints. At each candidate point x, a quadratic approximation to the criterion faunction is computed using the value of gradient at point x. A penalty term is incorporated to render optimizaion convex and conservative. The algorithm is "globally convergent" in the sense that it is guaranteed to con- verge to a local optimum from any feasible starting point. The implementation is based on CCSA algorithm described in :cite:`Svanberg2002`. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_mma .. code-block:: "nlopt_mma" Minimize a scalar function using the method of moving asymptotes (MMA). The implementation is based on an algorithm described in :cite:`Svanberg2002`. The algorithm performs gradient based local optimization with equality (but not inequality) constraints. At each candidate point x, an approximation to the criterion faunction is computed using the value of gradient at point x. A quadratic penalty term is incorporated to render optimizaion convex and conservative. The algorithm is "globally convergent" in the sense that it is guaranteed to con- verge to a local optimum from any feasible starting point. - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_var .. code-block:: "nlopt_var" Minimize a scalar function limited memory switching variable-metric method. The algorithm relies on saving only limited number M of past updates of the gradient to approximate the inverse hessian. The large is M, the more memory is consumed Detailed explanation of the algorithm, including its two variations of rank-2 and rank-1 methods can be found in the following paper :cite:`Vlcek2006` . - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **rank_1_update** (bool): Whether I rank-1 or rank-2 update is used. ``` ```{eval-rst} .. dropdown:: nlopt_slsqp .. code-block:: "nlopt_slsqp" Optimize a scalar function based on SLSQP method. SLSQP solves gradient based nonlinearly constrained optimization problems. The algorithm treats the optimization problem as a sequence of constrained least-squares problems. The implementation is based on the procedure described in :cite:`Kraft1988` and :cite:`Kraft1994` . - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_direct .. code-block:: "nlopt_direct" Optimize a scalar function based on DIRECT method. DIRECT is the DIviding RECTangles algorithm for global optimization, described in :cite:`Jones1993` . Variations of the algorithm include locally biased routines (distinguished by _L suffix) that prove to be more efficients for functions that have few local minima. See the following for the DIRECT_L variant :cite:`Gablonsky2001` . Locally biased algorithms can be implmented both with deterministic and random (distinguished by _RAND suffix) search algorithm. Finally, both original and locally biased variants can be implemented with and without the rescaling of the bound constraints. Boolean arguments `locally_biased`, 'random_search', and 'unscaled_bouds' can be set to `True` or `False` to determine which method is run. The comprehensive list of available methods are: - "DIRECT" - "DIRECT_L" - "DIRECT_L_NOSCAL" - "DIRECT_L_RAND" - "DIRECT_L_RAND_NOSCAL" - "DIRECT_RAND" - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **locally_biased** (bool): Whether the "L" version of the algorithm is selected. - **random_search** (bool): Whether the randomized version of the algorithm is selected. - **unscaled_bounds** (bool): Whether the "NOSCAL" version of the algorithm is selected. ``` ```{eval-rst} .. dropdown:: nlopt_esch .. code-block:: "nlopt_esch" Optimize a scalar function using the ESCH algorithm. ESCH is an evolutionary algorithm that supports bound constraints only. Specifi cally, it does not support nonlinear constraints. More information on this method can be found in :cite:`DaSilva2010` , :cite:`DaSilva2010a` , :cite:`Beyer2002` and :cite:`Vent1975` . - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_isres .. code-block:: "nlopt_isres" Optimize a scalar function using the ISRES algorithm. ISRES is an implementation of "Improved Stochastic Evolution Strategy" written for solving optimization problems with non-linear constraints. The algorithm is supposed to be a global method, in that it has heuristics to avoid local minima. However, no convergence proof is available. The original method and a refined version can be found, respecively, in :cite:`PhilipRunarsson2005` and :cite:`Thomas2000` . - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. ``` ```{eval-rst} .. dropdown:: nlopt_crs2_lm .. code-block:: "nlopt_crs2_lm" Optimize a scalar function using the CRS2_LM algorithm. This implementation of controlled random search method with local mutation is based on :cite:`Kaelo2006` . The original CRS method is described in :cite:`Price1978` and :cite:`Price1983` . CRS class of algorithms starts with random population of points and evolves the points "randomly". The size of the initial population can be set via the param- meter population_size. If the user doesn't specify a value, it is set to the nlopt default of 10*(n+1). - **convergence.xtol_rel** (float): Stop when the relative movement between parameter vectors is smaller than this. - **convergence.xtol_abs** (float): Stop when the absolute movement between parameter vectors is smaller than this. - **convergence.ftol_rel** (float): Stop when the relative improvement between two iterations is smaller than this. - **convergence.ftol_abs** (float): Stop when the change of the criterion function between two iterations is smaller than this. - **stopping.maxfun** (int): If the maximum number of function evaluation is reached, the optimization stops but we do not count this as convergence. - **population_size** (int): Size of the population. If None, it's set to be 10 * (number of parameters + 1). ``` ## Optimizers from iminuit optimagic supports the [IMINUIT MIGRAD Optimizer](https://iminuit.readthedocs.io/). To use MIGRAD, you need to have [the iminuit package](https://github.com/scikit-hep/iminuit) installed (`pip install iminuit`). ```{eval-rst} .. dropdown:: iminuit_migrad **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.iminuit_migrad(stopping_maxfun=10_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="iminuit_migrad", algo_options={"stopping_maxfun=10_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.iminuit_migrad.IminuitMigrad ``` ## Nevergrad Optimizers optimagic supports following algorithms from the [Nevergrad](https://facebookresearch.github.io/nevergrad/index.html) library. To use these optimizers, you need to have [the nevergrad package](https://github.com/facebookresearch/nevergrad) installed. (`pip install nevergrad`).\ Two algorithms from nevergrad are not available in optimagic.\ `SPSA (Simultaneous Perturbation Stochastic Approximation)` - This is WIP in nevergrad and hence imprecise.\ `AXP (AX-platfofm)` - Very slow and not recommended. ```{eval-rst} .. dropdown:: nevergrad_pso **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_pso(stopping_maxfun=1_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_pso", algo_options={"stopping_maxfun": 1_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradPSO ``` ```{eval-rst} .. dropdown:: nevergrad_cmaes **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_cmaes(stopping_maxfun=1_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_cmaes", algo_options={"stopping_maxfun": 1_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradCMAES ``` ```{eval-rst} .. dropdown:: nevergrad_oneplusone **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_oneplusone(stopping_maxfun=1_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_oneplusone", algo_options={"stopping_maxfun": 1_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradOnePlusOne ``` ```{eval-rst} .. dropdown:: nevergrad_de **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_de(population_size="large", ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_de", algo_options={"population_size": "large", ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradDifferentialEvolution ``` ```{eval-rst} .. dropdown:: nevergrad_bo .. note:: Using this optimizer requires the `bayes-optim` package to be installed as well. This can be done with `pip install bayes-optim`. **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_bo(stopping_maxfun=1_000, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_bo", algo_options={"stopping_maxfun": 1_000, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradBayesOptim ``` ```{eval-rst} .. dropdown:: nevergrad_emna **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_emna(noise_handling=False, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_emna", algo_options={"noise_handling": False, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradEMNA ``` ```{eval-rst} .. dropdown:: nevergrad_cga **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_cga(stopping_maxfun=10_000) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_cga", algo_options={"stopping_maxfun": 10_000} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradCGA ``` ```{eval-rst} .. dropdown:: nevergrad_eda **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_eda(stopping_maxfun=10_000) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_eda", algo_options={"stopping_maxfun": 10_000} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradEDA ``` ```{eval-rst} .. dropdown:: nevergrad_tbpsa **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_tbpsa(noise_handling=False, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_tbpsa", algo_options={"noise_handling": False, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradTBPSA ``` ```{eval-rst} .. dropdown:: nevergrad_randomsearch **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_randomsearch(opposition_mode="quasi", ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_randomsearch", algo_options={"opposition_mode": "quasi", ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradRandomSearch ``` ```{eval-rst} .. dropdown:: nevergrad_samplingsearch **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.nevergrad_samplingsearch(sampler="Hammersley", scrambled=True) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_samplingsearch", algo_options={"sampler": "Hammersley", "scrambled": True} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradSamplingSearch ``` ```{eval-rst} .. dropdown:: nevergrad_wizard **How to use this algorithm:** .. code-block:: import optimagic as om from optimagic.optimizers.nevergrad_optimizers import Wizard om.minimize( ..., algorithm=om.algos.nevergrad_wizard(optimizer= Wizard.NGOptRW, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_wizard", algo_options={"optimizer": "NGOptRW", ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradWizard .. autoclass:: optimagic.optimizers.nevergrad_optimizers.Wizard ``` ```{eval-rst} .. dropdown:: nevergrad_portfolio **How to use this algorithm:** .. code-block:: import optimagic as om from optimagic.optimizers.nevergrad_optimizers import Portfolio om.minimize( ..., algorithm=om.algos.nevergrad_portfolio(optimizer= Portfolio.BFGSCMAPlus, ...) ) or .. code-block:: om.minimize( ..., algorithm="nevergrad_portfolio", algo_options={"optimizer": "BFGSCMAPlus", ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.nevergrad_optimizers.NevergradPortfolio .. autoclass:: optimagic.optimizers.nevergrad_optimizers.Portfolio ``` ## Bayesian Optimization We wrap the [BayesianOptimization](https://github.com/bayesian-optimization/BayesianOptimization) package. To use it, you need to have [bayesian-optimization](https://pypi.org/project/bayesian-optimization/) installed. Note: This optimizer requires `bayesian_optimization > 2.0.0` to be installed which is incompatible with `nevergrad > 1.0.3`. ```{eval-rst} .. dropdown:: bayes_opt **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.bayes_opt(n_iter=50, ...) ) or .. code-block:: om.minimize( ..., algorithm="bayes_opt", algo_options={"n_iter": 50, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.bayesian_optimizer.BayesOpt ``` ## Gradient Free Optimizers Optimizers from the [gradient_free_optimizers](https://github.com/SimonBlanke/Gradient-Free-Optimizers?tab=readme-ov-file) package are available in optimagic. To use it, you need to have [gradient_free_optimizers](https://pypi.org/project/gradient_free_optimizers) installed. ```{eval-rst} .. dropdown:: gfo_hillclimbing **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_hillclimbing(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_hillclimbing", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOHillClimbing :members: :inherited-members: Algorithm, object ``` ```{eval-rst} .. dropdown:: gfo_stochastichillclimbing **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_stochastichillclimbing(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_stochastichillclimbing", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOStochasticHillClimbing :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_repulsinghillclimbing **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_repulsinghillclimbing(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_repulsinghillclimbing", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFORepulsingHillClimbing :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_simulatedannealing **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_simulatedannealing(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_simulatedannealing", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOSimulatedAnnealing :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_downhillsimplex **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_downhillsimplex(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_downhillsimplex", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFODownhillSimplex :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_powells_method **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_powells_method(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_powells_method", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOPowellsMethod :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_pso **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm=om.algos.gfo_pso(stopping_maxiter=1_000, ...), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=[1.0, 2.0, 3.0], algorithm="gfo_pso", algo_options={"stopping_maxiter": 1_000, ...}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOParticleSwarmOptimization :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_parallel_tempering **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm=om.algos.gfo_parallel_tempering(population_size=15, n_iter_swap=5), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm="gfo_parallel_tempering", algo_options={"population_size": 15, "n_iter_swap": 5}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOParallelTempering :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_spiral_optimization **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm=om.algos.gfo_spiral_optimization(population_size=15, decay_rate=0.95), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm="gfo_spiral_optimization", algo_options={"population_size": 15, "decay_rate": 0.95}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOSpiralOptimization :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_genetic_algorithm **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm=om.algos.gfo_genetic_algorithm(population_size=20, mutation_rate=0.6), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm="gfo_genetic_algorithm", algo_options={"population_size": 20, "mutation_rate": 0.6}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOGeneticAlgorithm :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_evolution_strategy **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm=om.algos.gfo_evolution_strategy(population_size=15, crossover_rate=0.4), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm="gfo_evolution_strategy", algo_options={"population_size": 15, "crossover_rate": 0.4}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFOEvolutionStrategy :members: :inherited-members: Algorithm, object :member-order: bysource ``` ```{eval-rst} .. dropdown:: gfo_differential_evolution **How to use this algorithm.** .. code-block:: python import optimagic as om import numpy as np om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm=om.algos.gfo_differential_evolution(population_size=20, mutation_rate=0.8), bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) or using the string interface: .. code-block:: python om.minimize( fun=lambda x: x @ x, params=np.array([1.0, 2.0, 3.0]), algorithm="gfo_differential_evolution", algo_options={"population_size": 20, "mutation_rate": 0.8}, bounds = om.Bounds(lower = np.array([1,1,1]), upper=np.array([5,5,5])) ) **Description and available options:** .. autoclass:: optimagic.optimizers.gfo_optimizers.GFODifferentialEvolution :members: :inherited-members: Algorithm, object :member-order: bysource ``` ## Pygad Optimizer We wrap the pygad optimizer. To use it you need to have [pygad](https://pygad.readthedocs.io/en/latest/) installed. ```{eval-rst} .. dropdown:: pygad **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.pygad(num_generations=100, ...) ) or .. code-block:: om.minimize( ..., algorithm="pygad", algo_options={"num_generations": 100, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.pygad_optimizer.Pygad ``` ## PySwarms Optimizers optimagic supports the following continuous algorithms from the [PySwarms](https://pyswarms.readthedocs.io/en/latest/) library: (GlobalBestPSO, LocalBestPSO, GeneralOptimizerPSO). To use these optimizers, you need to have [the pyswarms package](https://github.com/ljvmiranda921/pyswarms) installed. (`pip install pyswarms`). ```{eval-rst} .. dropdown:: pyswarms_global_best **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.pyswarms_global_best(n_particles=50, ...) ) or .. code-block:: om.minimize( ..., algorithm="pyswarms_global_best", algo_options={"n_particles": 50, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.pyswarms_optimizers.PySwarmsGlobalBestPSO :members: :inherited-members: Algorithm, object ``` ```{eval-rst} .. dropdown:: pyswarms_local_best **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.pyswarms_local_best(n_particles=50, k_neighbors=3, ...) ) or .. code-block:: om.minimize( ..., algorithm="pyswarms_local_best", algo_options={"n_particles": 50, "k_neighbors": 3, ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.pyswarms_optimizers.PySwarmsLocalBestPSO :members: :inherited-members: Algorithm, object ``` ```{eval-rst} .. dropdown:: pyswarms_general **How to use this algorithm:** .. code-block:: import optimagic as om om.minimize( ..., algorithm=om.algos.pyswarms_general(n_particles=50, topology_type="star", ...) ) or .. code-block:: om.minimize( ..., algorithm="pyswarms_general", algo_options={"n_particles": 50, "topology_type": "star", ...} ) **Description and available options:** .. autoclass:: optimagic.optimizers.pyswarms_optimizers.PySwarmsGeneralPSO :members: :inherited-members: Algorithm, object ``` ## References ```{eval-rst} .. bibliography:: refs.bib :labelprefix: algo_ :filter: docname in docnames :style: unsrt ``` (estimagic)= # Estimagic *estimagic* is a subpackage of *optimagic* that helps you to fit nonlinear statistical models to data and perform inference on the estimated parameters. As a user, you need to code up the objective function that defines the estimator. This is either a likelihood (ML) function or a Method of Simulated Moments (MSM) objective function. Everything else is done by *estimagic*. Everything else means: - Optimize your objective function - Calculate asymptotic or bootstrapped standard errors and confidence intervals - Create publication quality tables - Perform sensitivity analysis on MSM models `````{grid} 1 2 2 2 --- gutter: 3 --- ````{grid-item-card} :text-align: center :img-top: ../_static/images/light-bulb.svg :class-img-top: index-card-image :shadow: md ```{button-link} tutorials/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Tutorials ``` New users of estimagic should read this first. ```` ````{grid-item-card} :text-align: center :img-top: ../_static/images/books.svg :class-img-top: index-card-image :shadow: md ```{button-link} explanation/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- Explanations ``` Background information on key topics central to the package. ```` ````{grid-item-card} :text-align: center :columns: 12 :img-top: ../_static/images/coding.svg :class-img-top: index-card-image :shadow: md ```{button-link} reference/index.html --- click-parent: ref-type: ref class: stretched-link index-card-link sd-text-primary --- API Reference ``` Detailed description of the estimagic API. ```` ````` ```{toctree} --- hidden: true maxdepth: 1 --- tutorials/index explanation/index reference/index ``` # Estimagic Tutorials Estimagic hast functions to estimate the parameters of maximum likelihood or simulation models. You provide a likelihood or moment simulation function. Estimagic produces parameter estimates and standard errors in a format that can be easily used to create publication quality latex or html tables. ```{toctree} --- maxdepth: 1 --- likelihood_overview msm_overview bootstrap_overview estimation_tables_overview ``` { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Likelihood estimation\n", "\n", "This notebook shows how to do a simple maximum likelihood (ml) estimation with estimagic. As an illustrating example, we implement a simple linear regression model. This is the same example model used as in the method of moments notebook.\n", "\n", "We proceed in 4 steps:\n", "\n", "\n", "1. Create a data generating process\n", "2. Set up a likelihood function\n", "3. Maximize the likelihood function\n", "4. Calculate standard errors, confidence intervals, and p-values\n", "\n", "The user only needs to do step 1 and 2. The rest is done by `estimate_ml`. \n", "\n", "To be very clear: Estimagic is not a package to estimate linear models or other models that are implemented in Stata, statsmodels or anywhere else. Its purpose is to estimate parameters with custom likelihood or method of simulated moments functions. We just use an ordered logit model as an example of a very simple likelihood function.\n", "\n", "\n", "## Model:\n", "\n", "$$ y = \\beta_0 + \\beta_1 x + \\epsilon, \\text{ where } \\epsilon \\sim N(0, \\sigma^2)$$\n", "\n", "We aim to estimate $\\beta_0, \\beta_1, \\sigma^2$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "from scipy.stats import norm\n", "\n", "import estimagic as em\n", "\n", "rng = np.random.default_rng(seed=0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1. Create a data generating process" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def simulate_data(params, n_draws):\n", " x = rng.normal(0, 1, size=n_draws)\n", " e = rng.normal(0, params.loc[\"sd\", \"value\"], size=n_draws)\n", " y = params.loc[\"intercept\", \"value\"] + params.loc[\"slope\", \"value\"] * x + e\n", " return pd.DataFrame({\"y\": y, \"x\": x})" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "true_params = pd.DataFrame(\n", " data=[[2, -np.inf], [-1, -np.inf], [1, 1e-10]],\n", " columns=[\"value\", \"lower_bound\"],\n", " index=[\"intercept\", \"slope\", \"sd\"],\n", ")\n", "true_params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data = simulate_data(true_params, n_draws=100)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Define the `loglike` function" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def normal_loglike(params, data):\n", " norm_rv = norm(\n", " loc=params.loc[\"intercept\", \"value\"] + params.loc[\"slope\", \"value\"] * data[\"x\"],\n", " scale=params.loc[\"sd\", \"value\"],\n", " )\n", " contributions = norm_rv.logpdf(data[\"y\"])\n", " return {\"contributions\": contributions, \"value\": contributions.sum()}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "A few remarks before we move on:\n", "\n", "1. There are numerically better ways to calculate the likelihood; we chose this implementation for brevity and readability. \n", "2. The loglike function takes params and other arguments. You are completely flexible with respect to the number and names of the other arguments as long as the first argument is params. \n", "3. The loglike function returns a dictionary with the entries \"contributions\" and \"value\". The \"contributions\" are the log likelihood evaluations of each individual in the dataset. The \"value\" are their sum. The \"value\" entry could be omitted, the \"contributions\" entry, however, is mandatory. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Estimate the model" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "start_params = true_params.assign(value=[100, 100, 100])\n", "\n", "res = em.estimate_ml(\n", " loglike=normal_loglike,\n", " params=start_params,\n", " optimize_options={\"algorithm\": \"scipy_lbfgsb\"},\n", " loglike_kwargs={\"data\": data},\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.summary().round(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 4. What's in the results?\n", "\n", "`LikelihoodResult` objects provide attributes and methods to calculate standard errors, confidence intervals, and p-values. For all three, several methods are available. You can even calculate cluster robust standard errors. \n", "\n", "A few examples are:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.params" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.cov(method=\"robust\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "res.se()" ] } ], "metadata": { "kernelspec": { "display_name": "optimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" } }, "nbformat": 4, "nbformat_minor": 2 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Bootstrap Tutorial\n", "\n", "This notebook contains a tutorial on how to use the bootstrap functionality provided by estimagic. We start with the simplest possible example of calculating standard errors and confidence intervals for an OLS estimator without as well as with clustering. Then we progress to more advanced examples.\n", "\n", "In the example here, we will work with the \"exercise\" example dataset taken from the seaborn library.\n", "\n", "The working example will be a linear regression to investigate the effects of exercise time on pulse." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import seaborn as sns\n", "import statsmodels.api as sm\n", "\n", "import estimagic as em" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Prepare the dataset" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = sns.load_dataset(\"exercise\", index_col=0)\n", "replacements = {\"1 min\": 1, \"15 min\": 15, \"30 min\": 30}\n", "df = df.replace({\"time\": replacements})\n", "df[\"constant\"] = 1\n", "\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Doing a very simple bootstrap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The first thing we need is a function that calculates the bootstrap outcome, given an empirical or re-sampled dataset. The bootstrap outcome is the quantity for which you want to calculate standard errors and confidence intervals. In most applications those are just parameter estimates.\n", "\n", "In our case, we want to regress \"pulse\" on \"time\" and a constant. Our outcome function looks as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def ols_fit(data):\n", " y = data[\"pulse\"]\n", " x = data[[\"constant\", \"time\"]]\n", " params = sm.OLS(y, x).fit().params\n", "\n", " return params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In general, the user-specified outcome function may return any pytree (e.g. numpy.ndarray, pandas.DataFrame, dict etc.). In the example here, it returns a pandas.Series." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we are ready to calculate confidence intervals and standard errors." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results_without_cluster = em.bootstrap(data=df, outcome=ols_fit)\n", "results_without_cluster.ci()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results_without_cluster.se()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above function call represents the minimum that a user has to specify, making full use of the default options, such as drawing a 1_000 bootstrap draws, using the \"percentile\" bootstrap confidence interval, not making use of parallelization, etc.\n", "\n", "If, for example, we wanted to take 10_000 draws, while parallelizing on two cores, and using a \"bc\" type confidence interval, we would simply call the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results_without_cluster2 = em.bootstrap(\n", " data=df, outcome=ols_fit, n_draws=10_000, n_cores=2\n", ")\n", "\n", "results_without_cluster2.ci(ci_method=\"bc\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Doing a clustered bootstrap" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the cluster robust variant of the bootstrap, the original dataset is divided into clusters according to the values of some user-specified variable, and then clusters are drawn uniformly with replacement in order to create the different bootstrap samples. \n", "\n", "In order to use the cluster robust boostrap, we simply specify which variable to cluster by. In the example we are working with, it seems sensible to cluster on individuals, i.e. on the column \"id\" of our dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "results_with_cluster = em.bootstrap(data=df, outcome=ols_fit, cluster_by=\"id\")\n", "\n", "results_with_cluster.se()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the estimated standard errors are indeed of smaller magnitude when we use the cluster robust bootstrap. \n", "\n", "Finally, we can compare our bootstrap results to a regression on the full sample using statsmodels' OLS function.\n", "We see that the cluster robust bootstrap yields standard error estimates very close to the ones of the cluster robust regression, while the regular bootstrap seems to overestimate the standard errors of both coefficients.\n", "\n", "**Note**: We would not expect the asymptotic statsmodels standard errors to be exactly the same as the bootstrapped standard errors.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "y = df[\"pulse\"]\n", "x = df[[\"constant\", \"time\"]]\n", "\n", "\n", "cluster_robust_ols = sm.OLS(y, x).fit(cov_type=\"cluster\", cov_kwds={\"groups\": df[\"id\"]})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Splitting up the process" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In many situations, the above procedure is enough. However, sometimes it may be important to split the bootstrapping process up into smaller steps. Examples for such situations are:\n", "\n", "1. You want to look at the bootstrap estimates\n", "2. You want to do a bootstrap with a low number of draws first and add more draws later without duplicated calculations\n", "3. You have more bootstrap outcomes than just the parameters\n", "\n", "### 1. Accessing bootstrap outcomes\n", "\n", "The bootstrap outcomes are stored in the results object you get back when calling the bootstrap function. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result = em.bootstrap(data=df, outcome=ols_fit, seed=1234)\n", "my_outcomes = result.outcomes\n", "\n", "my_outcomes[:5]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To further compare the cluster bootstrap to the uniform bootstrap, let's plot the sampling distribution of the parameters on time. We can again see that the standard error is smaller when we cluster on the subject id. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "result_clustered = em.bootstrap(data=df, outcome=ols_fit, seed=1234, cluster_by=\"id\")\n", "my_outcomes_clustered = result_clustered.outcomes" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# clustered distribution in blue\n", "sns.histplot(\n", " pd.DataFrame(my_outcomes_clustered)[\"time\"], kde=True, stat=\"density\", linewidth=0\n", ")\n", "\n", "# non-clustered distribution in orange\n", "sns.histplot(\n", " pd.DataFrame(my_outcomes)[\"time\"],\n", " kde=True,\n", " stat=\"density\",\n", " linewidth=0,\n", " color=\"orange\",\n", ");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Calculating standard errors and confidence intervals from existing bootstrap result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you've already run ``bootstrap`` once, you can simply pass the existing result object to a new call of ``bootstrap``. Estimagic reuses the existing bootstrap outcomes and now only draws ``n_draws`` - ``n_existing`` outcomes instead of drawing entirely new ``n_draws``. Depending on the ``n_draws`` you specified (this is set to 1_000 by default), this may save considerable computation time. \n", "\n", "We can go on and compute confidence intervals and standard errors, just the same way as before, with several methods (e.g. \"percentile\" and \"bc\"), yet without duplicated evaluations of the bootstrap outcome function. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "my_results = em.bootstrap(\n", " data=df,\n", " outcome=ols_fit,\n", " existing_result=result,\n", ")\n", "my_results.ci(ci_method=\"t\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can use this to calculate confidence intervals with several methods (e.g. \"percentile\" and \"bc\") without duplicated evaluations of the bootstrap outcome function." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 2. Extending bootstrap results with more draws\n", "\n", "It is often the case that, for speed reasons, you set the number of bootstrap draws quite low, so you can look at the results earlier and later decide that you need more draws. \n", "\n", "As an example, we will take an initial sample of 500 draws. We then extend it with another 1500 draws. \n", "\n", "*Note*: It is very important to use a different random seed when you calculate the additional outcomes!!!" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "initial_result = em.bootstrap(data=df, outcome=ols_fit, seed=5471, n_draws=500)\n", "initial_result.ci(ci_method=\"t\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "combined_result = em.bootstrap(\n", " data=df, outcome=ols_fit, existing_result=initial_result, seed=2365, n_draws=2000\n", ")\n", "combined_result.ci(ci_method=\"t\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 3. Using less draws than totally available bootstrap outcomes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You have a large sample of bootstrap outcomes but want to compute summary statistics only on a subset? No problem! Estimagic got you covered. You can simply pass any number of ``n_draws`` to your next call of ``bootstrap``, regardless of the size of the existing sample you want to use. We already covered the case where ``n_draws`` > ``n_existing`` above, in which case estimagic draws the remaining bootstrap outcomes for you.\n", "\n", "If ``n_draws`` <= ``n_existing``, estimagic takes a random subset of the existing outcomes - and voilà! " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "subset_result = em.bootstrap(\n", " data=df, outcome=ols_fit, existing_result=combined_result, seed=4632, n_draws=500\n", ")\n", "subset_result.ci(ci_method=\"t\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Accessing the bootstrap samples" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also possible to just access the bootstrap samples. You may do so, for example, if you want to calculate your bootstrap outcomes in parallel in a way that is not yet supported by estimagic (e.g. on a large cluster or super-computer)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from estimagic.bootstrap_samples import get_bootstrap_samples\n", "\n", "rng = np.random.default_rng(1234)\n", "my_samples = get_bootstrap_samples(data=df, rng=rng)\n", "my_samples[0]" ] } ], "metadata": { "kernelspec": { "display_name": "estimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" }, "vscode": { "interpreter": { "hash": "e8a16b1bdcc80285313db4674a5df2a5a80c75795379c5d9f174c7c712f05b3a" } } }, "nbformat": 4, "nbformat_minor": 4 } { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# How to generate publication quality tables\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Estimagic can create publication quality tables of parameter estimates in LaTeX or HTML. It works with the results from `estimate_ml` and `estimate_msm` but also supports statsmodels results out of the box. \n", "\n", "You can get almost limitless flexibility if you split the table generation into two steps. The fist generates a DataFrame which you can customize to your liking, the second renders that DataFrame in LaTeX or HTML. If you are interested in this feature, search for \"render_inputs\" below." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "# Make necessary imports\n", "import pandas as pd\n", "import statsmodels.formula.api as sm\n", "from IPython.core.display import HTML\n", "\n", "import estimagic as em\n", "from estimagic.config import EXAMPLE_DIR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Create tables from statsmodels results" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv(EXAMPLE_DIR / \"diabetes.csv\", index_col=0)\n", "mod1 = sm.ols(\"target ~ Age + Sex\", data=df).fit()\n", "mod2 = sm.ols(\"target ~ Age + Sex + BMI + ABP\", data=df).fit()\n", "models = [mod1, mod2]" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	target
	(1)	(2)
Intercept	152.00$^{*** }$	152.00$^{*** }$
	(3.61)	(2.85)
Age	301.00$^{*** }$	37.20$^{ }$
	(77.10)	(64.10)
Sex	17.40$^{ }$	-107.00$^{* }$
	(77.10)	(62.10)
BMI		787.00$^{*** }$
		(65.40)
ABP		417.00$^{*** }$
		(69.50)
\n", "
Observations	442	442
R$^2$	0.04	0.40
Adj. R$^2$	0.03	0.40
Residual Std. Error	75.90	60
F Statistic	8.06$^{***}$	72.90$^{***}$
\n", "
Note:	^*p<0.01; ^p<0.05; ^*p<0.1

" ], "text/plain": [ "" ] }, "execution_count": 26, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(em.estimation_table(models, return_type=\"html\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Adding estimagic results\n", "\n", "`estimate_ml` and `estimate_msm` can both generate summaries of estimation results. Those summaries are either DataFrames with the columns `\"value\"`, `\"standard_error\"`, `\"p_value\"` and `\"stars\"` or pytrees containing such DataFrames. \n", "\n", "For examples, check out our tutorials on [`estimate_ml`](likelihood_overview.ipynb) and [`estimate_msm`](msm_overview.ipynb).\n", "\n", "\n", "Assume we got the following DataFrame from an estimation summary:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

	value	standard_error	p_value
Intercept	142.123	3.14150	1.000000e-08
Age	51.456	2.71828	1.000000e-08
Sex	-33.789	1.61800	1.000000e-08

\n", "

" ], "text/plain": [ " value standard_error p_value\n", "Intercept 142.123 3.14150 1.000000e-08\n", "Age 51.456 2.71828 1.000000e-08\n", "Sex -33.789 1.61800 1.000000e-08" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "params = pd.DataFrame(\n", " {\n", " \"value\": [142.123, 51.456, -33.789],\n", " \"standard_error\": [3.1415, 2.71828, 1.6180],\n", " \"p_value\": [1e-8] * 3,\n", " },\n", " index=[\"Intercept\", \"Age\", \"Sex\"],\n", ")\n", "params" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can either use just the params DataFrame or a dictionary containing \"params\" and additional information in `estimation_table`." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "mod3 = {\"params\": params, \"name\": \"target\", \"info\": {\"n_obs\": 445}}\n", "models = [mod1, mod2, mod3]" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	target
	(1)	(2)	(3)
Intercept	152.00$^{*** }$	152.00$^{*** }$	142.00$^{*** }$
	(3.61)	(2.85)	(3.14)
Age	301.00$^{*** }$	37.20$^{ }$	51.50$^{*** }$
	(77.10)	(64.10)	(2.72)
Sex	17.40$^{ }$	-107.00$^{* }$	-33.80$^{*** }$
	(77.10)	(62.10)	(1.62)
BMI		787.00$^{*** }$
		(65.40)
ABP		417.00$^{*** }$
		(69.50)
\n", "
Observations	442	442	445
R$^2$	0.04	0.40
Adj. R$^2$	0.03	0.40
Residual Std. Error	75.90	60
F Statistic	8.06$^{***}$	72.90$^{***}$
\n", "
Note:	^*p<0.01; ^p<0.05; ^*p<0.1

" ], "text/plain": [ "" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(em.estimation_table(models, return_type=\"html\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Selecting the right return_type\n", "\n", "The following return types are supported:\n", "- `\"latex\"`: Returns a string that you can save and import into a LaTeX document\n", "- `\"html\"`: Returns a string that you can save and import into a HTML document.\n", "- `\"render_inputs\"`: Returns a dictionary with the following entries:\n", " - `\"body\"`: A DataFrame containing the main table\n", " - `\"footer\"`: A DataFrame containing the statisics\n", " - other stuff that you should ignore\n", "- `\"dataframe\"`: Returns a DataFrame you can look at in a notebook" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Use `render_inputs` for maximum flexibility\n", "\n", "As an example, let's assume we want to remove a few rows from the footer.\n", "\n", "Let's first look at the footer we get from `estimation_table`" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "

\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	target
	(1)	(2)	(3)
Observations	442	442	445
R$^2$	0.04	0.40
Adj. R$^2$	0.03	0.40
Residual Std. Error	75.90	60
F Statistic	8.06$^{***}$	72.90$^{***}$

\n", "

" ], "text/plain": [ " target \n", " (1) (2) (3)\n", "Observations 442 442 445\n", "R$^2$ 0.04 0.40 \n", "Adj. R$^2$ 0.03 0.40 \n", "Residual Std. Error 75.90 60 \n", "F Statistic 8.06$^{***}$ 72.90$^{***}$ " ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "render_inputs = em.estimation_table(models, return_type=\"render_inputs\")\n", "footer = render_inputs[\"footer\"]\n", "footer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can remove the rows we don't need and render it to html. " ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	target
	(1)	(2)	(3)
Intercept	152.00$^{*** }$	152.00$^{*** }$	142.00$^{*** }$
	(3.61)	(2.85)	(3.14)
Age	301.00$^{*** }$	37.20$^{ }$	51.50$^{*** }$
	(77.10)	(64.10)	(2.72)
Sex	17.40$^{ }$	-107.00$^{* }$	-33.80$^{*** }$
	(77.10)	(62.10)	(1.62)
BMI		787.00$^{*** }$
		(65.40)
ABP		417.00$^{*** }$
		(69.50)
\n", "
R$^2$	0.04	0.40
Observations	442	442	445
\n", "
Note:	^*p<0.01; ^p<0.05; ^*p<0.1

" ], "text/plain": [ "" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "render_inputs[\"footer\"] = footer.loc[[\"R$^2$\", \"Observations\"]]\n", "HTML(em.render_html(**render_inputs))" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Using this 2-step-procedure, we can also easily add additional rows to the footer.\n", "\n", "Note that we add the row using `.loc[(\"Statsmodels\", )]` since the index of `render_inputs[\"footer\"]` is a MultiIndex.\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

	target
	(1)	(2)	(3)
Intercept	152.00$^{*** }$	152.00$^{*** }$	142.00$^{*** }$
	(3.61)	(2.85)	(3.14)
Age	301.00$^{*** }$	37.20$^{ }$	51.50$^{*** }$
	(77.10)	(64.10)	(2.72)
Sex	17.40$^{ }$	-107.00$^{* }$	-33.80$^{*** }$
	(77.10)	(62.10)	(1.62)
BMI		787.00$^{*** }$
		(65.40)
ABP		417.00$^{*** }$
		(69.50)
\n", "
R$^2$	0.04	0.40
Observations	442	442	445
Statsmodels	Yes	Yes	No
\n", "
Note:	^*p<0.01; ^p<0.05; ^*p<0.1

" ], "text/plain": [ "" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "render_inputs[\"footer\"].loc[(\"Statsmodels\",)] = [\"Yes\"] * 2 + [\"No\"]\n", "HTML(em.render_html(**render_inputs))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Advanced options \n", "\n", "Below is an exmample that demonstrates how to use advanced options to customize your table." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [], "source": [ "stats_dict = {\n", " \"n_obs\": \"Observations\",\n", " \"rsquared\": \"R$^2$\",\n", " \"rsquared_adj\": \"Adj. R$^2$\",\n", " \"resid_std_err\": \"Residual Std. Error\",\n", " \"fvalue\": \"F Statistic\",\n", " \"show_dof\": True,\n", "}" ] }, { "cell_type": "code", "execution_count": 34, "metadata": {}, "outputs": [ { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "

Table Latex(render_latex(**render_inputs))Title
	Dependent variable: target
	Model 1	Model 2	Model 3
Constant	152.133$^{*** }$	152.133$^{*** }$	142.123$^{*** }$
	(3.610)	(2.853)	(3.142)
Age	301.161$^{*** }$	37.241$^{ }$	51.456$^{*** }$
	(77.060)	(64.117)	(2.718)
Gender	17.392$^{ }$	-106.578$^{* }$	-33.789$^{*** }$
	(77.060)	(62.125)	(1.618)
BMI		787.179$^{*** }$
		(65.424)
ABP		416.674$^{*** }$
		(69.495)
\n", "
Observations	442	442	445
R$^2$	0.035	0.400
Adj. R$^2$	0.031	0.395
Residual Std. Error	75.888(df=439)	59.976(df=437)
F Statistic	8.059$^{***}$(df=2;439)	72.913$^{***}$(df=4;437)
\n", "
Note:	^*p<0.01; ^p<0.05; ^*p<0.1

" ], "text/plain": [ "" ] }, "execution_count": 34, "metadata": {}, "output_type": "execute_result" } ], "source": [ "HTML(\n", " em.estimation_table(\n", " models=models,\n", " return_type=\"html\",\n", " custom_param_names={\"Intercept\": \"Constant\", \"Sex\": \"Gender\"},\n", " custom_col_names=[\"Model 1\", \"Model 2\", \"Model 3\"],\n", " custom_col_groups={\"target\": \"Dependent variable: target\"},\n", " render_options={\"caption\": \"Table Latex(render_latex(**render_inputs))Title\"},\n", " stats_options=stats_dict,\n", " number_format=\"{0:.3f}\",\n", " )\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***Note 1***: You can pass a dictionary for `custom_col_names` to rename specific columns, e.g. `custom_col_names={\"(1)\": \"Model 1\"}`, leaving names of the other columns at default values.\n", "\n", "***Note 2***: In addition to renaming the default column groups by passing a dictionary for `custom_col_groups`, you can also pass a list to create custom column groups, e.g. `custom_col_groups=[\"target\", \"target\", \"not target\"]` will group the first two columns under the name `\"target\"`, and the last column under the name `\"not target\"`.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## LaTeX peculiarities\n", "\n", "By default, tables in `render_latex` are structured in compliance with `siunitx` package. This is done by setting column formats to `S` in the default rendering options defined internally. \n", "To get nicely formatted tables, you need to add the following to your LaTeX preamble:\n", "```latex\n", "\\usepackage{siunitx}\n", "\\sisetup{\n", " input-symbols = (),\n", " table-align-text-post = false\n", " group-digits = false,\n", " }\n", "```\n", "The first line in `\\sisetup` is necessary if you have parentheses in your table cells (e.g. when displaying standard errors or confidence intervals), otherwise LaTex will raise an error.\n", "\n", "The second argument is necessary so that there is no spacing between the significance stars and the numerical values.\n", "\n", "The third line prevents digits in numbers being grouped into groups of threes, which is the default behaviour.\n", "This line is optional, but recommended.\n", "\n", "By default, whenever calling `render_latex`, a warning will be raised about this. To silence the warning, set `siunitx_warning=False` in the relvant function calls (when calling `estimation_table` with `return_type=tex` or when calling `render_latex`)\n", "\n", "If you don't want to generate `siunitx` style tables, you can pass `render_options={\"column_format\":}` to your function calls. \n", "\n", "You can influence the format of the output table with keyword arguments passed via `render_options`. For the list of supported keyword arguments see [the documentation of pandas.io.formats.style.Styler.to_latex](https://pandas.pydata.org/docs/reference/api/pandas.io.formats.style.Styler.to_latex.html)\n", "\n", "\n", "\n", "By default, `siunitx` will center table columns around the decimal point. This means, that if there is a number in a column that has many comparatively larger number of symbols after the decimal point (e.g. when there is a number with scientific notation), there will be extra spacing between that column and the preceeding one, since there is as much space reserved for the column before the decimal point, as there is after it. \n", "\n", "You can adjust the spacing between columns, by using the format `S[table-format =x.y]` for the numeric columns, where `x` and `y` control the space pre and post the decimal point, respecitvely. We further show a case with the described problem and the solution to that problem. For number with scientific notations, use `S[table-format=x.yez]`, where `y` reserves the space for the exponential, and `z` reserves the space for the column after the decimal point.\n", "\n", "Compiling the following LaTex table will result in extra spacing between columns `(2)` and `(3)`:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```latex\n", "\n", "\\begin{tabular}{lSSS}\n", " \\toprule\n", " & \\multicolumn{3}{c}{target} \\\\\n", " \\cmidrule(lr){2-4}\n", "\n", " & (1) & (2) & (3) \\\\\n", " \\midrule\n", " Intercept & 152.00$^{*** }$ & 152.00$^{*** }$ & 1.43e08$^{*** }$ \\\\\n", " & (3.61) & (2.85) & (3.14) \\\\\n", " Age & 301.00$^{*** }$ & 37.20$^{ }$ & 51.50$^{*** }$ \\\\\n", " & (77.10) & (64.10) & (2.72) \\\\\n", " Sex & 17.40$^{ }$ & -107.00$^{* }$ & -33.80$^{*** }$ \\\\\n", " & (77.10) & (62.10) & (1.62) \\\\\n", " BMI & & 787.00$^{*** }$ & \\\\\n", " & & (65.40) & \\\\\n", " ABP & & 417.00$^{*** }$ & \\\\\n", " & & (69.50) & \\\\\n", " \\midrule\n", " R$^2$ & 0.04 & 0.40 & \\\\\n", " Observations & \\multicolumn{1}{c}{442} & \\multicolumn{1}{c}{442} & \\multicolumn{1}{c}{445} \\\\\n", " \\midrule\n", " \\textit{Note:} & \\multicolumn{3}{r}{$^{***}$p$<$0.01;$^{**}$p$<$0.05;$^{*}$p$<$0.1} \\\\\n", " \\bottomrule\n", "\\end{tabular}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can get a nicer output by setting the format of the last column to, for example, `S[table-format=3.2e4]`, via passing `render_options={'column_format':'lSSS[table-format = 3.2e4]'}`. The resulting table of `render_latex` will look like the following:" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "```latex\n", "\n", "\\begin{tabular}{lSSS[table-format = 3.2e4]}\n", " \\toprule\n", " & \\multicolumn{3}{c}{target} \\\\\n", " \\cmidrule(lr){2-4}\n", "\n", " & (1) & (2) & (3) \\\\\n", " \\midrule\n", " Intercept & 152.00$^{*** }$ & 152.00$^{*** }$ & 1.43e08$^{*** }$ \\\\\n", " & (3.61) & (2.85) & (3.14) \\\\\n", " Age & 301.00$^{*** }$ & 37.20$^{ }$ & 51.50$^{*** }$ \\\\\n", " & (77.10) & (64.10) & (2.72) \\\\\n", " Sex & 17.40$^{ }$ & -107.00$^{* }$ & -33.80$^{*** }$ \\\\\n", " & (77.10) & (62.10) & (1.62) \\\\\n", " BMI & & 787.00$^{*** }$ & \\\\\n", " & & (65.40) & \\\\\n", " ABP & & 417.00$^{*** }$ & \\\\\n", " & & (69.50) & \\\\\n", " \\midrule\n", " R$^2$ & 0.04 & 0.40 & \\\\\n", " Observations & \\multicolumn{1}{c}{442} & \\multicolumn{1}{c}{442} & \\multicolumn{1}{c}{445} \\\\\n", " \\midrule\n", " \\textit{Note:} & \\multicolumn{3}{r}{$^{***}$p$<$0.01;$^{**}$p$<$0.05;$^{*}$p$<$0.1} \\\\\n", " \\bottomrule\n", "\\end{tabular}\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [] } ], "metadata": { "@webio": { "lastCommId": null, "lastKernelId": null }, "interpreter": { "hash": "5cdb9867252288f10687117449de6ad870b49795ca695c868016dc0022895cce" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.10" } }, "nbformat": 4, "nbformat_minor": 4 } { "cells": [ { "cell_type": "markdown", "id": "0", "metadata": {}, "source": [ "# Method of Simulated Moments (MSM)\n", "\n", "This tutorial shows you how to do a Method of Simulated Moments estimation in estimagic. The Method of Simulated Moments (MSM) is a nonlinear estimation principle that is very useful for fitting complicated models to the data. The only ingredient required is a function that simulates the model outcomes you observe in some empirical dataset. \n", "\n", "In the tutorial here, we will use a simple linear regression model. This is the same model which we use in the tutorial on maximum likelihood estimation.\n", "\n", "Throughout the tutorial, we only talk about MSM estimation. However, the more general case of indirect inference estimation works exactly the same way. \n", "\n", "\n", "## Steps of MSM estimation\n", "\n", "1. Load (simulate) empirical data \n", "2. Define a function to calculate estimation moments on the data \n", "3. Calculate the covariance matrix of the empirical moments (with ``get_moments_cov``)\n", "4. Define a function to simulate moments from the model \n", "5. Estimate the model, calculate standard errors, do sensitivity analysis (with ``estimate_msm``)\n", "\n", "## Example: Estimate the parameters of a regression model\n", "\n", "The model we consider here is a simple regression model with only one explanatory variable (plus a constant). The goal is to estimate the slope coefficients and the error variance from a simulated data set.\n", "\n", "The estimation mechanics are exactly the same for more complicated models. A model is always defined by a function that can take parameters (here: the mean, variance and lower_cutoff and upper_cutoff) and returns a number of simulated moments (mean, variance, soft_min and soft_max of simulated exam points).\n", "\n", "### Model:\n", "\n", "$$ y = \\beta_0 + \\beta_1 x + \\epsilon, \\text{ where } \\epsilon \\sim N(0, \\sigma^2)$$\n", "\n", "We aim to estimate $\\beta_0, \\beta_1, \\sigma^2$." ] }, { "cell_type": "code", "execution_count": null, "id": "1", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import plotly.io as pio\n", "\n", "pio.renderers.default = \"notebook_connected\"\n", "\n", "import estimagic as em\n", "\n", "rng = np.random.default_rng(seed=0)" ] }, { "cell_type": "markdown", "id": "2", "metadata": {}, "source": [ "## 1. Simulate data" ] }, { "cell_type": "code", "execution_count": null, "id": "3", "metadata": {}, "outputs": [], "source": [ "def simulate_data(params, n_draws, rng):\n", " x = rng.normal(0, 1, size=n_draws)\n", " e = rng.normal(0, params.loc[\"sd\", \"value\"], size=n_draws)\n", " y = params.loc[\"intercept\", \"value\"] + params.loc[\"slope\", \"value\"] * x + e\n", " return pd.DataFrame({\"y\": y, \"x\": x})" ] }, { "cell_type": "code", "execution_count": null, "id": "4", "metadata": {}, "outputs": [], "source": [ "true_params = pd.DataFrame(\n", " data=[[2, -np.inf], [-1, -np.inf], [1, 1e-10]],\n", " columns=[\"value\", \"lower_bound\"],\n", " index=[\"intercept\", \"slope\", \"sd\"],\n", ")\n", "\n", "data = simulate_data(true_params, n_draws=100, rng=rng)" ] }, { "cell_type": "markdown", "id": "5", "metadata": {}, "source": [ "## 2. Calculate Moments" ] }, { "cell_type": "code", "execution_count": null, "id": "6", "metadata": {}, "outputs": [], "source": [ "def calculate_moments(sample):\n", " moments = {\n", " \"y_mean\": sample[\"y\"].mean(),\n", " \"x_mean\": sample[\"x\"].mean(),\n", " \"yx_mean\": (sample[\"y\"] * sample[\"x\"]).mean(),\n", " \"y_sqrd_mean\": (sample[\"y\"] ** 2).mean(),\n", " \"x_sqrd_mean\": (sample[\"x\"] ** 2).mean(),\n", " }\n", " return pd.Series(moments)" ] }, { "cell_type": "code", "execution_count": null, "id": "7", "metadata": {}, "outputs": [], "source": [ "empirical_moments = calculate_moments(data)\n", "empirical_moments" ] }, { "cell_type": "markdown", "id": "8", "metadata": {}, "source": [ "## 3. Calculate the covariance matrix of empirical moments\n", "\n", "The covariance matrix of the empirical moments (``moments_cov``) is needed for three things:\n", "1. to calculate the weighting matrix\n", "2. to calculate standard errors\n", "3. to calculate sensitivity measures\n", "\n", "We will calculate ``moments_cov`` via a bootstrap. Depending on your problem, there can be other ways to calculate the covariance matrix." ] }, { "cell_type": "code", "execution_count": null, "id": "9", "metadata": {}, "outputs": [], "source": [ "moments_cov = em.get_moments_cov(\n", " data, calculate_moments, bootstrap_kwargs={\"n_draws\": 5_000, \"seed\": 0}\n", ")\n", "\n", "moments_cov" ] }, { "cell_type": "markdown", "id": "10", "metadata": {}, "source": [ "``get_moments_cov`` mainly just calls estimagic's bootstrap function. See our [bootstrap_tutorial](bootstrap_overview.ipynb) for background information. \n", "\n" ] }, { "cell_type": "markdown", "id": "11", "metadata": {}, "source": [ "## 4. Define a function to calculate simulated moments\n", "\n", "In a real world application, this is the step that would take most of the time. However, in our very simple example, all the work is already done by numpy." ] }, { "cell_type": "code", "execution_count": null, "id": "12", "metadata": {}, "outputs": [], "source": [ "def simulate_moments(params, n_draws=10_000, seed=0):\n", " rng = np.random.default_rng(seed)\n", " sim_data = simulate_data(params, n_draws, rng)\n", " sim_moments = calculate_moments(sim_data)\n", " return sim_moments" ] }, { "cell_type": "code", "execution_count": null, "id": "13", "metadata": {}, "outputs": [], "source": [ "simulate_moments(true_params)" ] }, { "cell_type": "markdown", "id": "14", "metadata": {}, "source": [ "## 5. Estimate the model parameters\n", "\n", "Estimating a model consists of the following steps:\n", "\n", "- Building a criterion function that measures a distance between simulated and empirical moments\n", "- Minimizing this criterion function\n", "- Calculating the Jacobian of the model\n", "- Calculating standard errors, confidence intervals and p-values\n", "- Calculating sensitivity measures\n", "\n", "This can all be done in one go with the ``estimate_msm`` function. This function has sensible default values, so you only need a minimum number of inputs. However, you can configure almost any aspect of the workflow via optional arguments. If you need even more control, you can call the lower level functions, which the now famliliar``estimate_msm`` function is built on, directly. " ] }, { "cell_type": "code", "execution_count": null, "id": "15", "metadata": {}, "outputs": [], "source": [ "start_params = true_params.assign(value=[100, 100, 100])\n", "\n", "res = em.estimate_msm(\n", " simulate_moments,\n", " empirical_moments,\n", " moments_cov,\n", " start_params,\n", " optimize_options=\"scipy_lbfgsb\",\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "16", "metadata": {}, "outputs": [], "source": [ "res.summary()" ] }, { "cell_type": "markdown", "id": "17", "metadata": {}, "source": [ "## What's in the result?\n", "\n", "`MomentsResult` objects provide attributes and methods to calculate standard errors, confidence intervals and p-values. For all three, several methods are available. You can even calculate cluster robust standard errors.\n", "\n", "A few examples are:" ] }, { "cell_type": "code", "execution_count": null, "id": "18", "metadata": {}, "outputs": [], "source": [ "res.params" ] }, { "cell_type": "code", "execution_count": null, "id": "19", "metadata": {}, "outputs": [], "source": [ "res.cov(method=\"robust\")" ] }, { "cell_type": "code", "execution_count": null, "id": "20", "metadata": {}, "outputs": [], "source": [ "res.se()" ] }, { "cell_type": "markdown", "id": "21", "metadata": {}, "source": [ "## How to visualize sensitivity measures?" ] }, { "cell_type": "code", "execution_count": null, "id": "22", "metadata": {}, "outputs": [], "source": [ "from estimagic import lollipop_plot\n", "\n", "sensitivity_data = res.sensitivity(kind=\"bias\").abs().T\n", "\n", "fig = lollipop_plot(sensitivity_data)\n", "\n", "fig = fig.update_layout(height=500, width=900)\n", "fig.show()" ] } ], "metadata": { "kernelspec": { "display_name": "estimagic", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.14" }, "vscode": { "interpreter": { "hash": "e8a16b1bdcc80285313db4674a5df2a5a80c75795379c5d9f174c7c712f05b3a" } } }, "nbformat": 4, "nbformat_minor": 5 } # Explanation ```{toctree} --- maxdepth: 1 --- bootstrap_ci bootstrap_montecarlo_comparison cluster_robust_likelihood_inference ``` (bootstrap-cis)= # Bootstrap Confidence Intervals We use the notation and formulations provided in chapter 10 of {cite}`Hansen2020`. The first supported confidence interval type is the **"percentile"** confidence interval, as discussed in section 10.10 of the Hansen textbook. Let $\{ \hat{\theta}_1^*, ..., \hat{\theta}_B^*\}$ denote the estimates of estimator $\hat{\theta}$ for the B bootstrap samples. The idea of the percentile confidence interval is to simply take the empirical quantiles $q_{p}^*$ of this distributions, so we have $$ CI^{percentile} = [q_{\alpha/2}^*, q_{1-\alpha/2}^*]. $$ The second supported confidence interval **"normal"** is based on a normal approximation and discussed in Hansen's section 10.9. Let $s_{boot}$ be the sample standard error of the distribution of bootstrap estimators, $z_q$ the q-quantile of a standard normal distribution and $\hat{\theta}$ be the full sample estimate of $\theta$. Then, the asymptotic normal confidence interval is given by $$ CI^{normal} = [\hat{\theta} - z_{1- \alpha/2} s_{boot}, \hat{\theta} + z_{1- \alpha/2} s_{boot}]. $$ The bias-corrected **"bc"** bootstrap confidence interval addresses the issue of biased estimators. This problem is often present when estimating nonlinear models. Econometric details are discussed in section 10.17 of Hansen. Let $$ p^* = \frac{1}{B} \sum_{b=1}^B 1(\hat{\theta}_b^* \leq \hat{\theta}) $$ and define $z_0^* = \Phi^{-1} (p^*)$, where $\Phi$ is the standard normal cdf. The bias correction works via correcting the significance level. Define $x(\alpha) = \Phi(z_\alpha + 2 z_0^*)$ as the corrected significance level for a target significant level of $\alpha$. Then, the bias-corrected confidence interval is given by $$ CI^{bc} = [q_{x(\alpha/2)}^*, q_{x(1-\alpha/2)}^*]. $$ A further refined version of the bias-corrected confidence interval is the bias-corrected and accelerated interval, short **"bca"**, as discussed in section 10.20 of Hansen. The general idea is to correct for skewness sampling distribution. Downsides of this confidence interval are that it takes quite a lot of time to compute, since it features calculating leave-one-out estimates of the original sample. Formally, again, the significance levels are adjusted. Define $$ \hat{a}=\frac{\sum_{i=1}^{n}\left(\bar{\theta}-\hat{\theta}_{(-i)}\right)^{3}} {6\left(\sum_{i=1}^{n}\left(\bar{\theta}-\hat{\theta}_{(-i)}\right)^{2} \right)^{3 / 2}}, $$ where $\bar{\theta}=\frac{1}{n} \sum_{i=1}^{n} \widehat{\theta}_{(-i)}$. This is an estimator for the skewness of $\hat{\theta}$. Then, the corrected significance level is given by $$ x(\alpha)=\Phi(z_{0}+\frac{z_{\alpha}+z_{0}}{1-a(z_{\alpha}+z_{0})}) $$ and the bias-corrected and accelerated confidence interval is given by $$ CI^{bca} = [q_{x(\alpha/2)}^*, q_{x(1-\alpha/2)}^*]. $$ The studentized confidence interval, here called **"t"** type confidence interval first studentizes the bootstrap parameter distribution, i.e. applies the transformation $\frac{\hat{\theta}_b-\hat{\theta}}{s_{boot}}$, and then builds the confidence interval based on the estimated quantile function of the studentized data $\hat{G}$: $$ CI^{t} = \left[\hat{\theta}+\hat{\sigma} \hat{G}^{-1}(\alpha / 2), \hat{\theta}+\hat{\sigma} \hat{G}^{-1}(1-\alpha / 2)\right] $$ The final supported confidence interval method is the **"basic"** bootstrap confidence interval, which is derived in section 3.4 of {cite}`Wassermann2006`, where it is called the pivotal confidence interval. It is given by $$ CI^{basic} = \left[\hat{\theta}+\left(\hat{\theta}-\hat{\theta}_{u}^{\star}\right), \hat{\theta}+\left(\hat{\theta}-\hat{\theta}_{l}^{\star}\right)\right], $$ where $\hat{\theta}_{u}^{\star}$ denotes the $1-\alpha/2$ empirical quantile of the bootstrap estimate distribution for parameter $\theta$ and $\hat{\theta}_{l}^{\star}$ denotes the $\alpha/2$ quantile. ```{eval-rst} .. bibliography:: ../../refs.bib :filter: docname in docnames ``` { "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Bootstrap Monte Carlo Comparison" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In this juypter notebook, we perform a Monte Carlo exercise to illustrate the importance of using the cluster robust variant of the bootstrap when data within clusters is correlated. \n", "\n", "The main idea is to repeatedly draw clustered samples, get both uniform and clustered bootstrap estimates in these samples, and then compare how often the true null hypothesis is rejected." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Generating Process" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The true data generating process is given by\n", "\n", "$$ logit(y_{i,g}) = \\beta_0 + \\beta_1 (x_{i,g}) + \\epsilon_{i,g}, $$\n", "\n", "where the independent variable $x_{i,g} = x_i + x_g$ and the noise term $\\epsilon_{i,g} = \\epsilon_i + \\epsilon_g$ each consist of an individual and a cluster term.\n", "\n", "In the simulations we perform below, we have $\\beta_0 = \\beta_1 =0$. $x_i$ and $x_g$ are drawn from a standard normal distribution, and $\\epsilon_i$ and $\\epsilon_g$ are drawn from a normal distribution with $\\mu_0$ and $\\sigma=0.5$. The value of $\\sigma$ is chosen to not blow up rejection rates in the independent case too much." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import pandas as pd\n", "import scipy\n", "import statsmodels.api as sm\n", "from joblib import Parallel, delayed\n", "\n", "import estimagic as em" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "def create_clustered_data(nclusters, nobs_per_cluster, true_beta=0):\n", " \"\"\"Create a bivariate clustered dataset with specified number of\n", " clusters and number of observations per cluster that has a population\n", " value of true_beta for the logit coefficient on the independent variable.\n", "\n", " Args:\n", " nclusters (int): Number of clusters.\n", " nobs_per_cluster (int): Number of observations per cluster.\n", " true_beta (int): The true logit coefficient on x.\n", "\n", " Returns:\n", " pd.DataFrame: Clustered dataset.\n", " \"\"\"\n", " x_cluster = np.random.normal(size=nclusters)\n", " x_ind = np.random.normal(size=nobs_per_cluster * nclusters)\n", " eps_cluster = np.random.normal(size=nclusters, scale=0.5)\n", " eps_ind = np.random.normal(size=nobs_per_cluster * nclusters, scale=0.5)\n", "\n", " y = []\n", " x = []\n", " cluster = []\n", "\n", " for g in range(nclusters):\n", " for i in range(nobs_per_cluster):\n", " key = (i + 1) * (g + 1) - 1\n", "\n", " arg = (\n", " true_beta * (x_cluster[g] + x_ind[key]) + eps_ind[key] + eps_cluster[g]\n", " )\n", "\n", " y_prob = 1 / (1 + np.exp(-arg))\n", " y.append(np.random.binomial(n=1, p=y_prob))\n", " x.append(x_cluster[g] + x_ind[(i + 1) * (g + 1) - 1])\n", " cluster.append(g)\n", "\n", " y = np.array(y)\n", " x = np.array(x)\n", " cluster = np.array(cluster)\n", "\n", " return pd.DataFrame({\"y\": y, \"x\": x, \"cluster\": cluster})" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Monte Carlo Simulation Code" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following function computes bootstrap t-values. As suggested my Cameron and Miller (2015), critical values are the 0.975 quantiles from a t distribution with `n_clusters` -1 degrees of freedom." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def get_t_values(data, sample_size=200, hyp_beta=0, cluster=False):\n", " \"\"\"Get bootstrap t-values for testing the hypothesis that beta == hyp_beta.\n", "\n", " Args:\n", " data (pd.DataFrame): Original dataset.\n", " sample_size (int): Number of bootstrap samples to draw.\n", " hyp_beta (float): Hypothesised value of beta.\n", " cluster (bool): Whether or not to cluster on the cluster column.\n", "\n", " Returns:\n", " float: T-Value of hypothesis.\n", " \"\"\"\n", "\n", " def logit_wrap(df):\n", " y = df[\"y\"]\n", " x = df[\"x\"]\n", "\n", " result = sm.Logit(y, sm.add_constant(x)).fit(disp=0).params\n", "\n", " return pd.Series(result, index=[\"constant\", \"x\"])\n", "\n", " if cluster is False:\n", " result = em.bootstrap(data=data, outcome=logit_wrap, n_draws=sample_size)\n", " estimates = pd.DataFrame(result.outcomes)[\"x\"]\n", "\n", " else:\n", " result = em.bootstrap(\n", " data=data,\n", " outcome=logit_wrap,\n", " n_draws=sample_size,\n", " cluster_by=\"cluster\",\n", " )\n", " estimates = pd.DataFrame(result.outcomes)[\"x\"]\n", "\n", " return (estimates.mean() - hyp_beta) / estimates.std()" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def monte_carlo(nsim, nclusters, nobs_per_cluster, true_beta=0, n_cores=8):\n", " \"\"\"Run a simulation for rejection rates and a logit data generating process.\n", "\n", " Rejection rates are based on a t distribution with nclusters-1 degrees of freedom.\n", "\n", " Args:\n", " nsim (int): Number of Monte Carlo draws.\n", " nclusters (int): Number of clusters in each generated dataset.\n", " nobs_per_cluster (int) Number of observations per cluster.\n", " true_beta (int): Population value of logit coefficient on x.\n", " n_cores (int): Number of jobs for Parallelization.\n", "\n", " Returns:\n", " pd.DataFrame: DataFrame of average rejection rates.\n", " \"\"\"\n", " np.zeros(nsim)\n", "\n", " np.zeros(nsim)\n", "\n", " def loop():\n", " df = create_clustered_data(nclusters, nobs_per_cluster, true_beta)\n", "\n", " return [get_t_values(df), get_t_values(df, cluster=True)]\n", "\n", " t_value_array = np.array(\n", " Parallel(n_jobs=n_cores)(delayed(loop)() for _ in range(nsim))\n", " )\n", " t_value_array = np.array([loop() for _ in range(nsim)])\n", "\n", " crit = scipy.stats.t.ppf(0.975, nclusters - 1)\n", "\n", " result = pd.DataFrame(np.abs(t_value_array) > crit, columns=[\"uniform\", \"cluster\"])\n", "\n", " return result" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we perform Monte Carlo simulations with the above functions. In each simulation, the sample size is 200, but the number of clusters varies across simulations. Be warned that the code below takes a long time to run." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "np.random.seed(505)\n", "\n", "results_list = []\n", "\n", "for g, k in [[20, 50], [100, 10], [500, 2]]:\n", " results_list.append(monte_carlo(nsim=100, nclusters=g, nobs_per_cluster=k))" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "mean_rejection_data = pd.DataFrame([x.mean() for x in results_list])\n", "mean_rejection_data[\"nclusters\"] = [20, 100, 500]\n", "mean_rejection_data.set_index(\"nclusters\", inplace=True)" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Text(0.5, 0.98, 'Comparison of Rejection Rates')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" }, { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsAAAAH2CAYAAAB+5DrCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAACI/UlEQVR4nOzdd1xTV/8H8E/YG1RARFAcOEARJ+JCXLharQutdba2ddRZV5+6+rR11121S619qtbVVmu11gEOcFGse6I4AEVliAqSnN8f95dATEAIgQTyeb9eeZHce+7NNwnoh8O558iEEAJERERERCbCzNAFEBERERGVJAZgIiIiIjIpDMBEREREZFIYgImIiIjIpDAAExEREZFJYQAmIiIiIpPCAExEREREJoUBmIiIiIhMCgMwEREREZkUBmCiEpCRkYGvvvoKoaGhqFixIqysrFCuXDkEBwdj5syZiI+PN3SJRs3HxwcymczQZRjEpk2b0LhxY9jZ2UEmk8HHx+e1xyjfr9w3R0dHNGzYEHPmzMHTp0/1Utvs2bMhk8mwfv16vZxPX27dugWZTIa2bdsauhQNbdu21fhs7O3t4efnh0mTJuHhw4eGLpHIJFgYugCisu748ePo3bs3EhMTYWdnh+bNm6NixYpITU3FqVOnEB0djQULFmD37t3o0KGDocslI3Lq1Cm88847sLGxQadOneDi4gJXV9cCH9+7d284ODhACIE7d+4gKioKs2fPxvbt23Hs2DE4OjoWY/XFZ/369Rg2bBhmzZqF2bNnG7ocnYSFhcHDwwMAkJCQgOjoaHz11VfYsmULTpw4gcqVKxfp/IcPH0ZoaCiGDBlidL+gEBkDBmCiYhQbG4v27dvjxYsXmDp1KmbMmAF7e3vVfoVCgV9//RVTpkzB3bt3DVipcTtw4ABevnxp6DJK3K5du6BQKLBixQoMHz680McvWrRIrcf42rVraNWqFc6dO4dly5bh008/LVJ9Y8aMQf/+/VGpUqUinUffKleujEuXLsHOzs7QpeRp2rRpaj3UCQkJaN++PS5duoRZs2bhu+++M1xxRCaAQyCIiokQAoMGDcKLFy8we/ZszJs3Ty38AoCZmRl69eqFM2fOoEmTJgaq1PjVqFEDderUMXQZJU75S1H16tX1cj5fX19MnDgRALBv374in8/V1RV16tSBs7Nzkc+lT5aWlqhTpw6qVKli6FIKrFKlSpg1axYA/Xw2RJQ/BmCiYrJ3716cP38eXl5e+M9//pNvW2dnZ9SrV09t27Nnz/Df//4X9erVg62tLZydndGmTRts3rxZ6zlyj5NdtWqV6rhq1aphwYIFEEIAAGJiYvDGG2+gfPnycHBwQI8ePXD79m2N8w0dOhQymQyHDx/Gn3/+iVatWsHBwQHlypVDr169cPnyZY1jXrx4ge+//x49evRA9erVYWtrCxcXl3zrzv08+/btQ2hoKFxcXCCTyZCSkqLx2nI7f/483nnnHVSvXh02NjZwc3NDYGAgxo8fj4SEBI32e/bsQceOHVGuXDnY2Nigdu3amDZtmup5css9vvXcuXN48803Ua5cOdjb2yMkJATHjx/X+nry8+jRI0yePBm+vr6wsbFB+fLl0blzZ/z1119q7davXw+ZTIZ169YBAEJDQ1XjRYv652x/f38AwIMHD7Tu37t3L7p16wY3NzdYW1ujevXqmDhxIh49eqTRNr8xwNnZ2Vi9ejWCg4Ph5OQEW1tbBAYGYunSpcjOztb63BkZGZg/fz6aNGkCJycn2Nvbo06dOhg9ejSuXr0KQBpDO2zYMADAnDlz1MbSKut43RjgjRs3olWrVnBycoKdnR0CAgIwd+5cvHjxQqNt7u/PyMhItGvXDo6OjnByckK3bt1w8eJFrc+hi/w+myNHjmDMmDEICAhAuXLlYGtrizp16mj9/h06dChCQ0MBABs2bFB7j14dMnLnzh2MGTMGNWrUUH1Pdu/ePc/v7+PHj6Nnz56oWrUqrK2t4eHhgWbNmmHatGl6G1tOVCIEERWL0aNHCwBiwoQJhT42LS1NNG7cWAAQbm5uok+fPqJLly7C2tpaABBjx47VOKZq1aoCgBg/frywtbUVXbt2Fd27dxeOjo4CgJg5c6Y4evSosLOzE40aNRL9+vUTNWvWFABEjRo1xLNnz9TON2TIEAFAjBo1SshkMtG0aVPRv39/4efnJwAIZ2dnERsbq3bMpUuXBADh6ekpQkNDRXh4uAgJCRGWlpYCgJg1a5ZG3crnGTFihNrzNG3aVKSkpKi9ttxOnz4tbGxsBAAREBAg+vXrJ7p3766q79ChQ2rtv/zySwFAWFhYiPbt24vw8HDh5eUlAIhatWqJxMREtfazZs0SAMTo0aOFnZ2dqF+/vggPDxcNGjQQAISNjY04d+5cQT9ScffuXVG9enUBQFSpUkWEh4eLdu3aCXNzcwFAfPXVV6q2R44cEUOGDBE1atQQAERYWJgYMmSIGDJkiDhy5Mhrn0v5fsXFxWns+/nnnwUA0apVK419U6dOFQCElZWVaNmypejTp4/w9fVVfY/k9R6tW7dObfuzZ89EaGioACDKly8vOnbsKN544w3h7u4uAIg333xTyOVytWPu378v/P39BQBRrlw58cYbb4g+ffqIRo0aCTMzM7FkyRIhhBBz584VLVu2FABEgwYNVO9L7vcmLi5OABAhISEar/H9999XfX5du3YVffr0Ea6urgKACA4OFhkZGWrtld+fEydOFObm5iIoKEj069dP1KpVSwAQFSpUEAkJCa/5RHKEhIRo/f4UQojjx48LAMLLy0tjX1BQkLCxsRHNmjUTvXv3Ft26dROVKlUSAIS/v79IT09Xtf32229FWFiY6nPL/R7t3LlT7fnKlSsnAIjatWuLXr16idatWwsLCwthbm4uNm/erFbD77//LszMzIRMJhNBQUGif//+onPnzqrvU23fb0TGigGYqJgo/5PeuHFjoY8dM2aMACBCQ0NFWlqaavulS5dUIWLXrl1qxyhDj6enp7h+/braMdbW1sLOzk74+PiI1atXq/ZlZmaKdu3aCQDihx9+UDuf8j9+AOKbb75RbVcoFKqgFBgYqHZMcnKy2L9/v1AoFGrbb968KXx8fISZmZnGf5K5n+fV/3BffW25DR48WAAQixYt0mh/6dIlcf/+fdXjkydPCjMzM+Hg4CCio6NV21+8eCH69u0rAIjevXurnUMZ7gCIZcuWqe0bP368ACAGDRqktV5tunfvLgCIt99+W2RmZqq2HzlyRNjZ2Qlzc3Pxzz//qB2jfG+0haX85BeAle/b559/rrb9l19+EQBEvXr1xLVr11TbFQqFmDlzpgAgwsPD1Y7JKwCPGjVK1V75S4wQ0i92Xbt2FQDUvg+FEKJ9+/YCgOjXr59amBNCCrRnz55VPV63bl2ev1Ap22sLwNu2bVP9jFy9elW1PSUlRbRq1UoAEJMmTVI7RvkZmJmZqYXH7Oxs0bt3bwFAzJgxQ2sd2uQXgJXv83vvvaexb8+ePWrvpRDS968y0M+ZM0dt36FDhwQAMWTIEK11pKamikqVKglzc3Px008/qe07deqUKFeunHBwcBAPHjxQbW/Tpo0AILZt26ZxvpMnT6r9W0Vk7BiAiYpJnTp1BACxd+/eQh339OlTYWtrK8zMzMSlS5c09i9fvlwAEB06dFDbrgw93333ncYxb731Vp69fr/99pvW/yiV//G3aNFC45isrCxV72lBeiSFkHqlAIjly5drfZ5u3brleay2ANylSxcBQKMXWhtl6Js+fbrGvqSkJNX7HR8fr9quDHctW7bUOCY5OVkAEFWrVn3tcwshxI0bNwQA4eDgIB49eqSxf+LEiVqDj74CsEKhELdv3xazZs1S9d69GjKVPdvaerUVCoUIDAwU5ubm4uHDh6rt2gJwUlKSsLS0FN7e3hp/VRBCiISEBGFlZSUCAgJU206cOCEACHd39wKFKF0DsDLArV27VuOYs2fPCplMJhwcHMTz589V25WfwcCBAzWOOX36dJ49zXnRFoDv378vVqxYIWxsbETNmjXVfnl7nWfPngkLCwvRqFEjte2vC8BLlizRGviVvvrqK42/TNStW1cA0AjiRKURxwATGZkzZ87g+fPnaNSokdYLvwYNGgQAOHbsGBQKhcb+Tp06aWxTXkSV3z5tY2YBoH///hrbLC0t0adPHwDS2MRXHT16FJ9//jlGjhyJYcOGYejQodi6dSsAaSYCbd58802t2/PSuHFjAMDo0aNx+PDhPMeV5q5x4MCBGvvc3d3RqVMnKBQKHDt2TGO/tvesQoUKKF++fJ7v2auOHj0KAOjcuTPKly+vsV/5mWp7L4uiWrVqkMlkMDMzQ9WqVTFnzhx07twZR44cgYODg6rdgwcPcPbsWfj6+mqMRQcAmUyGli1bQi6X48yZM/k+5+HDh/Hy5Ut07twZtra2Gvs9PDzg6+uLc+fO4fnz5wCAv//+GwAwYMCAYpua7eXLl4iOjgag/fsgICAAAQEBePr0KWJjYzX2a/s+qFWrFoC8f3byk3tct6enJz766CP4+fnhzJkzec6qce/ePaxZswbjx4/H8OHDMXToUIwcORJWVlZ5/lzlRTnuvFevXlr3t27dGgBw8uRJ1Tblz9ygQYNw6tQprf/+EJUWnAaNqJhUqFABAAo9sf39+/cBIM8FD1xcXODs7IzU1FQ8efJE9TxK2uYPVYad/PZlZmZqfb6qVatq3a6sT1kvAKSmpqJXr144ePCg1mMAID09Xev2wl6xP3nyZBw9elQ136mDgwOCg4PRrVs3DB06VG1mgte9p8rt9+7d09jn5eWl9RhHR0c8fvy4QLUW5fmLQjkPcFZWFq5cuYKYmBj8+eef+PLLL1UzDgDSRWOA9MvJ6xYcSU5Ozne/8lzffvstvv3223zbPn78GJUrV8adO3cASLN9FJdHjx4hKysLrq6uGrOxKPn4+ODs2bMF/j5QhvW8fnbyo5wHWC6XIy4uDsePH0dMTAzGjRunuvgxt6+++grTpk3T23SAys+pZcuW+bbL/Xl/+eWXOHfuHHbt2oVdu3ahXLlyaNWqFd58803VfNVEpQUDMFExCQwMxLFjxxATE4N33nlHr+fOL6SYmeX9h5389unD1KlTcfDgQYSEhGDOnDmoV68eXFxcYG5ujr/++gthYWGq2SheVdj/PJ2cnHDw4EEcO3YMu3btwuHDh3Hw4EHs378fc+fOxZEjR+Dr61ugc+n6fupLca1y9+o8wFu2bMGAAQPw2WefoXPnzggKCgIAVU+eh4cHwsLC8j1nXr8QKSnPFRgYiAYNGuTb1tra+nUvoUSV5PfBq/MAR0ZGIiwsDOvXr0e3bt1Uf2EBgOjoaEyaNAnOzs5YtmwZ2rZtCw8PD9X75+npWeheaOXn1KdPnzx/IQCg9lcob29vnD59GgcPHsTu3bsRERGhCsMLFixAVFSUxi/kRMaKAZiomHTr1g2rVq3C1q1bsWDBAlhYFOzHzdPTEwC0Tk0GSL2sKSkpsLW1Rbly5fRWb17yqkO5XVkvAOzcuRPm5ub4/fff4eTkpNb+5s2beq9NJpOhVatWaNWqFQDpT/njx4/Hpk2b8J///Ae//PKLqsa4uDjcvn0bfn5+GudR9oYVdfWtvLzuMy3u51cKDw/HwYMH8c0332D69Omqnnpl76arq2uRp1lTnqtVq1ZYsWJFgY7x9vYGANy4caNIz52fChUqwMrKCsnJycjIyNAa+krqc9CmTZs2mDlzJj755BN88skneOutt2Bubg5A+rkCgC+++AJDhgxRO+758+dITEws9PN5eXnhypUrmDZtmmpoQ0FYWFigU6dOqiEht2/fxvDhw3Hw4EHMnz8fCxYsKHQtRIbAMcBExaRz587w9/fH3bt38cUXX+TbNi0tDRcuXAAgjbOztbXFmTNntI7r++mnnwBIf7osid5JZYjMLTs7G9u3bwcAVfgEgCdPnsDJyUkj/OZ1Hn1zd3dXzXN6/vx51XbleMZNmzZpHPPw4UPs27dPNc61OCjfo71792qdc1j5mSrrLE6zZ8+GjY0NDh06pJrr1cvLC3Xq1MHFixdV8+3qKjQ0FObm5ti9e3eB/1yvXAJ806ZNBZpL1srKCgDyHff9KktLSzRv3hwAtM5Jff78eZw9exYODg4IDAws8Hn1afz48fDw8MC1a9ewZcsW1fYnT54A0D4MY+vWrVr/qvK696hjx44AcsK1rqpWrYqpU6cCUP+ZIzJ2DMBExUQmk+Gnn36CjY0NZs+ejenTpyMjI0OtjRACv//+O5o0aYJTp04BAOzt7TF8+HAoFAqMHj1a7ZirV6/i888/BwCMHTu2RF7H0aNH8cMPP6htmzVrFuLj4xEQEKAW2mrVqoUnT56o/ecNAEuWLMGhQ4f0WteaNWsQFxensX3Pnj0AcnoVAelCOTMzMyxfvhynT59Wbc/KysJHH32E58+fo1evXmrH6FP16tXRrVs3pKenY9y4cWrBMCoqCqtXr4a5uTlGjx5dLM+fW6VKlfDhhx8CgNovZjNmzIBCoUDv3r21XgT26NGj147pBaTe0+HDh+PWrVsYMGAAkpKSNNpcv35d9QsUADRr1gyhoaF48OAB3n//fY2fk1u3buHcuXOqx8oe9StXrry2ntw++ugjANIvAbn/IpGeno4xY8ZACIEPPvjAYGNZbW1tMW3aNADA3LlzVcFWebHd999/r/a9c/HiRVX4fNXr3qMPPvgA7u7uWLBgAb755huNC9qys7Oxb98+tVC7ZMkSrb3N2n7miIyeQeegIDIBR48eFRUrVhQAhJ2dnWjfvr14++23Rbdu3VTbbWxsxN9//606JvdCGO7u7qJv376ia9euqoUf8lsIQ5u85msVIu8po5TTP40cOVLIZDLRrFkzMWDAANViBU5OTiImJkbtmJ9++kk1d27r1q3FgAEDhJ+fnzAzMxMTJkzId7q1/Kb60vbalNN2+fn5id69e2ssUnH06FG19l988YVqIYwOHTqI/v37C29vbwFA+Pr6FniRh/xqys/du3dFtWrVVNOn9e/fX7Rv3161EMbixYs1jimOeYCFkKYis7W1FQDU5h7+5JNPVHPeNmrUSPTt21f06dNHNGzYUJibmwtnZ2e18+S3EEbHjh0FAGFvby9atmwpBgwYIN58803V4is9evTQeH9q166tWjzjzTffFH379tVYCEMIIZ4/f66aDzskJEQMGzZMvPvuu+LYsWNCiIIthGFrayu6desm+vbtK9zc3AQA0bx58zwXwsjrM1B+ngWV3zzAytemXODi119/FUJI0+55eHgIAKJatWqiX79+okOHDsLS0lL07ds3z+/FgIAAAUA0bdpUDB06VLz77rvit99+U+2PiopSLQLi7e0tunTpIt5++23Rrl074eLiIgCozX3s7OwszMzMRMOGDUW/fv1E3759VQuClC9fXm1uZSJjxwBMVALS09PFokWLREhIiHBzcxMWFhbCxcVFBAUFiVmzZok7d+5oHPP06VMxZ84c4efnJ6ytrYWjo6No1aqV+Pnnn7U+R3EF4EOHDoldu3aJ4OBgYWdnJ5ydnUWPHj3EhQsXtD7XH3/8IZo3by4cHR2Fi4uL6NChgzh8+HCe85LqGoB///13MXz4cOHv7y9cXFyEnZ2dqFWrlnjvvffE5cuXtZ5n9+7don379sLZ2VlYWVmJmjVriilTpojHjx8X6j3Lq6bXSU5OFpMmTRI1atQQVlZWwsXFRXTq1Ens27dPa/viCsBC5Mw93LdvX7XtERERom/fvsLT01NYWlqKChUqiICAADFmzBgRERGh1lb5Hq1fv17j/NnZ2WLDhg2iXbt2onz58sLS0lJ4enqK4OBgMWfOHHHlyhWNY9LS0sRnn30mAgIChK2trXBwcBB16tQRY8aMUVucQwhpsYaOHTsKZ2dnIZPJ1D6r/AKwEEL8+OOPokWLFsLBwUHY2NgIf39/8cUXX2idt7ikA7AQOXN9N23aVLXtzp074u233xaVK1cWNjY2om7dumLevHkiOzs7z+/Fa9euiZ49e4oKFSoIMzMzrXMnJyQkiClTpgh/f39hZ2cn7OzsRI0aNUSPHj3E+vXr1eaL/vHHH8Xbb78tateuLRwdHYWjo6Pw8/MTEydOFHfv3i3we0BkDGRC5HFJNhGZtKFDh2LDhg04dOiQ2tXqREpTp07FggUL8Msvv6Bv376GLoeIqMA4BpiIiHSiXBSjOOfvJSIqDgzARERUKDNmzECzZs1w4MAB1K1bFw0bNjR0SUREhcIATEREhbJ7925cvHgRXbp0wW+//VZsC3kQERUXjgEmIiIiIpPCHmAiIiIiMikMwERERERkUhiAiYiIiMikMAATERERkUlhACYiIiIik8IATEREREQmhQGYiIiIiEwKAzARERERmRQGYCIiIiIyKQzARERERGRSGICJiIiIyKQwABMRERGRSWEAJiIiIiKTwgBMRERERCaFAZiIiIiITAoDMBERERGZFAZgIiIiIjIpDMBEREREZFIYgImIiIjIpDAAExEREZFJYQAmIiIiIpPCAExEREREJoUBmIiIiIhMCgMwEREREZkUBmAiIiIiMikMwERERERkUhiAiYiIiMikMAATERERkUmxMHQBpYFCocD9+/fh6OgImUxm6HKIiIiI6BVCCKSnp8PT0xNmZvn38TIAF8D9+/fh7e1t6DKIiIiI6DXu3LkDLy+vfNswABeAo6MjAOkNdXJyMnA1RERERPSqtLQ0eHt7q3JbfhiAC0A57MHJyYkBmIiIiMiIFWS4Ki+CIyIiIiKTwgBMRERERCaFAZiIiIiITArHABMREZkQIQSys7Mhl8sNXQpRoZibm8PCwkIvU9IyABMREZmIrKwsJCQk4NmzZ4YuhUgndnZ2qFSpEqysrIp0HgZgIiIiE6BQKBAXFwdzc3N4enrCysqKiztRqSGEQFZWFh4+fIi4uDj4+vq+drGL/DAAExERmYCsrCwoFAp4e3vDzs7O0OUQFZqtrS0sLS1x+/ZtZGVlwcbGRudz8SI4IiIiE1KUXjMiQ9PX9y9/CoiIiIjIpDAAGxm5HDh8GNi0SfrKi3SJiIj0x8fHB0uXLlU9TkxMRMeOHWFvbw8XFxeD1UUliwHYiOzYAfj4AKGhwNtvS199fKTtRERExqKkO2vatm2L8ePHa2xfv359oUPrqVOn8P7776seL1myBAkJCYiNjcXVq1eLWKl+yGQy1c3CwgJVqlTBxIkTkZmZqdfnmT17NgIDAwt1zKu/QJRWvAjOSOzYAfTpAwihvv3ePWn7tm1Ar16GqY2IiEhpxw5g3Djg7t2cbV5ewLJlpeP/KTc3N7XHN27cQOPGjeHr66vzObOysoo8Lder1q1bh86dO+Ply5c4e/Yshg0bBnt7e/z3v//V6/MUB7lcDplMZtTjzY23MhMil0v/mLwafoGcbePHczgEEREZlrKzJnf4BXI6awz9F8uhQ4eiZ8+eWLRoESpVqoQKFSpg9OjRePnypapN7h5MHx8fbN++HT/++CNkMhmGDh0KAIiPj0ePHj3g4OAAJycn9OvXD0lJSapzKHtOv/vuO1SrVk01G4FMJsPatWvRvXt32NnZoW7duoiKisL169fRtm1b2Nvbo0WLFrhx48ZrX4uLiws8PDzg7e2N7t27o0ePHoiJiVFrs3r1atSoUQNWVlaoXbs2Nm7cqLY/v9exfv16zJkzB2fPnlX1Nq9fvx5CCMyePRtVqlSBtbU1PD09MXbsWABST/zt27cxYcIE1THKc7m4uOD333+Hn58frK2tER8fj1OnTqFjx45wdXWFs7MzQkJCNF6DTCbD6tWr0aVLF9ja2qJ69erYtm3ba9+fomIANgJHjmj+Y5KbEMCdO1I7IiIifRECyMgo2C0tDRg7Nv/OmnHjpHYFOZ+28+jDoUOHcOPGDRw6dAgbNmzA+vXrsX79eq1tT506hc6dO6Nfv35ISEjAsmXLoFAo0KNHDzx+/BgRERHYv38/bt68ifDwcLVjr1+/ju3bt2PHjh2IjY1Vbf/vf/+LwYMHIzY2FnXq1MHbb7+NDz74ANOnT8fp06chhMCYMWMK9ZquXr2KgwcPIigoSLVt586dGDduHCZNmoTz58/jgw8+wLBhw3Do0CEAeO3rCA8Px6RJk+Dv74+EhAQkJCQgPDwc27dvx5IlS7B27Vpcu3YNv/76K+rXrw8A2LFjB7y8vPDZZ5+pjlF69uwZ5s+fj++++w4XLlyAu7s70tPTMWTIEBw9ehTR0dHw9fVF165dkZ6ervb6ZsyYgd69e+Ps2bMYOHAg+vfvj0uXLhXqPSo0Qa+VmpoqAIjU1NRiOf/PPwsh/VOQ/+3nn4vl6YmIyAQ8f/5cXLx4UTx//ly17enTgv3/Uxy3p08LXntISIgYN26cxvZ169YJZ2dn1eMhQ4aIqlWriuzsbNW2vn37ivDwcNXjqlWriiVLlqge9+jRQwwZMkT1+K+//hLm5uYiPj5ete3ChQsCgDh58qQQQohZs2YJS0tL8eDBA7V6AIhPP/1U9TgqKkoAEN9//71q26ZNm4SNjU2+rxeAsLGxEfb29sLa2loAEN27dxdZWVmqNi1atBAjRoxQO65v376ia9euhXodDRo0UDvH4sWLRa1atdSeK7dX3z8hpM8BgIiNjc33dcnlcuHo6Ch27dql9lo//PBDtXZBQUFi5MiRWs+h7ftYqTB5jT3ARqBSJf22IyIiMlX+/v4wNzdXPa5UqRIePHhQ4OMvXboEb29veHt7q7b5+fnBxcVFrVeyatWqGuOJASAgIEB1v2LFigCg6kFVbnvx4gXS0tLyrWPJkiWIjY3F2bNnsXv3bly9ehWDBg1Sq7Nly5Zqx7Rs2VJVY0Ffx6v69u2L58+fo3r16hgxYgR27tyJ7OzsfGsFACsrK7XXDgBJSUkYMWIEfH194ezsDCcnJzx9+hTx8fFq7YKDgzUeF3cPMC+CMwKtW0sXENy7p/1PQjKZtL9165KvjYiIyi47O+Dp04K1jYwEunZ9fbs9e4A2bQr23AXl5OSE1NRUje0pKSlwdnZW22Zpaan2WCaTQaFQFPzJCsje3l7r9tzPrxwjq23b62ry8PBAzZo1AQC1a9dGeno6BgwYgM8//1y1vTh4e3vjypUr+Pvvv7F//36MGjUKCxcuREREhMZ7m5utra3G0tpDhgzBo0ePsGzZMlStWhXW1tYIDg5GVlZWsdVfUOwBNgLm5tLVs4AUdrVZulRqR0REpC8yGWBvX7Bbp05SZ0xe/0/JZIC3t9SuIOfL6zza1K5dW+PiKQCIiYlBrVq1dHz12tWtWxd37tzBnTt3VNsuXryIlJQU+Pn56fW5CkPZq/38+XMAUp3Hjh1Ta3Ps2DFVjQV5HVZWVpBrucLe1tYWb7zxBpYvX47Dhw8jKioK586dy/cYbY4dO4axY8eia9eu8Pf3h7W1NZKTkzXaRUdHazyuW7dugZ5DV+wBNhK9eklTnb06tQwAfPxx6ZhahoiIyi5lZ02fPlJ4zf0XS2WYLa7OmpEjR2LlypUYO3Ys3nvvPVhbW+OPP/7Apk2bsGvXLr0+V4cOHVC/fn0MHDgQS5cuRXZ2NkaNGoWQkBA0adJEr8+Vn5SUFCQmJkKhUODatWv47LPPUKtWLVUwnDx5Mvr164eGDRuiQ4cO2LVrF3bs2IG///67wK/Dx8cHcXFxiI2NhZeXFxwdHbFp0ybI5XIEBQXBzs4OP/30E2xtbVG1alXVMZGRkejfvz+sra3h6uqa52vw9fXFxo0b0aRJE6SlpWHy5MmwtbXVaLd161Y0adIErVq1wv/+9z+cPHkS33//vb7fUjXsATYivXoBt24Bhw4BP/8MvPOOtD0ioviuliUiIiooZWdN5crq2728ine++urVqyMyMhKXL19Ghw4dEBQUhF9++QVbt25F586d9fpcMpkMv/32G8qVK4c2bdqgQ4cOqF69OrZs2aLX53mdYcOGoVKlSvDy8sKAAQPg7++PP//8ExYWUt9lz549sWzZMixatAj+/v5Yu3Yt1q1bh7Zt2xb4dfTu3RudO3dGaGgo3NzcsGnTJri4uODbb79Fy5YtERAQgL///hu7du1ChQoVAACfffYZbt26hRo1amgdA53b999/jydPnqBRo0YYNGgQxo4dC3d3d412c+bMwebNmxEQEIAff/wRmzZtKvbedtn/X4FH+UhLS4OzszNSU1Ph5ORUYs/74AFQpQqQmSmF4IKMqSIiItLmxYsXiIuLU5u3VldyuTQ1Z0KCdIF269Ycpke6kclk2LlzJ3r27Fmg9vl9Hxcmr7EH2Ii5uwPDhkn3FywwbC1ERERK5uZA27bAgAHSV4ZfKm0YgI3cpEnS2Ko//gAuXDB0NURERESlHwOwkatZM2dM1aJFhq2FiIiISJ+EEAUe/qBPDMClwOTJ0tf//S//JZOJiIiI6PUYgEuBoCAgJAR4+TJnvmAiIiIi0g0DcCkxZYr0de1aQMtiOERERERUQAzApUSXLkC9ekB6uhSCiYiIiEg3DMClhEwmrQgHSCvtZGYatBwiIiKiUosBuBQZMEBafSchQbogjoiIiIgKjwG4FLGyAiZMkO4vXAgoFIath4iIyJjIZDL8+uuvhi6DSgEG4FJmxAjA2Rm4fBnYvdvQ1RARkUmSy4HDh4FNm6SvcnmxP2ViYiI++ugjVK9eHdbW1vD29sYbb7yBAwcOFMvzHT58GDKZDCkpKcVyfkAK7MqbhYUFqlSpgokTJyJTz+McZ8+ejcDAwEId4+Pjg6VLl+q1DmPCAFzKODkBI0dK9xcuNGwtRERkgnbsAHx8gNBQ4O23pa8+PtL2YnLr1i00btwYBw8exMKFC3Hu3Dns3bsXoaGhGD16dLE9rz4IIZCdnZ3n/nXr1iEhIQFxcXH4+uuvsXHjRnz++eclWKHu5HI5FKX0z9EMwKXQ2LHScIijR4Hjxw1dDRERmYwdO4A+fTRXZbp3T9peTCF41KhRkMlkOHnyJHr37o1atWrB398fEydORHR0tNZjtPXgxsbGQiaT4datWwCA27dv44033kC5cuVgb28Pf39/7NmzB7du3UJoaCgAoFy5cpDJZBg6dCgAQKFQYO7cuahWrRpsbW3RoEEDbNu2TeN5//zzTzRu3BjW1tY4evRonq/NxcUFHh4e8Pb2Rvfu3dGjRw/ExMSotVm9ejVq1KgBKysr1K5dGxs3blTbHx8fjx49esDBwQFOTk7o168fkpKSAADr16/HnDlzcPbsWVVv8/r16yGEwOzZs1GlShVYW1vD09MTY8eOBQC0bdsWt2/fxoQJE1THKM/l4uKC33//HX5+frC2tkZ8fDxOnTqFjh07wtXVFc7OzggJCdF4DTKZDKtXr0aXLl1ga2uL6tWrq71vJY0BuBSqVAkYNEi6z15gIiLSmRBARkbBbmlpUg+MENrPAwDjxkntCnI+befR4vHjx9i7dy9Gjx4Ne3t7jf0uLi46v/zRo0cjMzMTkZGROHfuHObPnw8HBwd4e3tj+/btAIArV64gISEBy/5/Jaq5c+fixx9/xJo1a3DhwgVMmDAB77zzDiIiItTOPW3aNMybNw+XLl1CQEBAgeq5evUqDh48iKCgINW2nTt3Yty4cZg0aRLOnz+PDz74AMOGDcOhQ4cASIG8R48eePz4MSIiIrB//37cvHkT4eHhAIDw8HBMmjQJ/v7+SEhIQEJCAsLDw7F9+3YsWbIEa9euxbVr1/Drr7+ifv36AIAdO3bAy8sLn332meoYpWfPnmH+/Pn47rvvcOHCBbi7uyM9PR1DhgzB0aNHER0dDV9fX3Tt2hXp6elqr2/GjBno3bs3zp49i4EDB6J///64dOlSYT4y/RH0WqmpqQKASE1NNXQpKpcuCQEIIZNJ94mIiPLz/PlzcfHiRfH8+fOcjU+fSv+ZGOL29GmB6j5x4oQAIHbs2PHatgDEzp07hRBCHDp0SAAQT548Ue3/559/BAARFxcnhBCifv36Yvbs2VrPpe34Fy9eCDs7O3H8+HG1tu+++64YMGCA2nG//vprgeq1sbER9vb2wtraWgAQ3bt3F1lZWao2LVq0ECNGjFA7rm/fvqJr165CCCH++usvYW5uLuLj41X7L1y4IACIkydPCiGEmDVrlmjQoIHaORYvXixq1aql9ly5Va1aVSxZskRt27p16wQAERsbm+/rksvlwtHRUezatUvttX744Ydq7YKCgsTIkSPzPdertH4f/7/C5DX2AJdSdeoAPXpI/4osXmzoaoiIiIqHKGBPsS7Gjh2Lzz//HC1btsSsWbPw77//5tv++vXrePbsGTp27AgHBwfV7ccff8SNGzfU2jZp0qRANSxZsgSxsbE4e/Ysdu/ejatXr2KQ8s+8AC5duoSWLVuqHdOyZUtVz+mlS5fg7e0Nb29v1X4/Pz+4uLjk27vat29fPH/+HNWrV8eIESOwc+fOfMcqK1lZWWn0aCclJWHEiBHw9fWFs7MznJyc8PTpU8THx6u1Cw4O1nhsqB5gBuBSbPJk6euPPwKJiYathYiISiE7O+Dp04Ld9uwp2Dn37CnY+ezsCnQ6X19fyGQyXL58uVAvzcxMiji5A/TLly/V2rz33nu4efMmBg0ahHPnzqFJkyZYsWJFnud8+vQpAOCPP/5AbGys6nbx4kWN8azahmto4+HhgZo1a6J27dro1q0b5syZgy1btuD69esFOl5X3t7euHLlCr7++mvY2tpi1KhRaNOmjcZ79CpbW1vVmGClIUOGIDY2FsuWLcPx48cRGxuLChUqICsrqzhfQpEwAJdiLVsCLVoAWVnA8uWGroaIiEodmQywty/YrVMnwMtLOiavc3l7S+0Kcr68zvOK8uXLIywsDKtWrUJGRobG/rymKXNzcwMAtfGrsbGxGu28vb3x4YcfYseOHZg0aRK+/fZbAFJPJyDNdKCU+8KvmjVrqt1y98AWhbm5OQDg+fPnAIC6devi2LFjam2OHTsGPz8/1f47d+7gzp07qv0XL15ESkqKqo2VlZXa61CytbXFG2+8geXLl+Pw4cOIiorCuXPn8j1Gm2PHjmHs2LHo2rUr/P39YW1tjeTkZI12r16wGB0djbp16xboOfTN6ALwqlWr4OPjAxsbGwQFBeHkyZN5tr1w4QJ69+4NHx8fyGQyrfPVzZ07F02bNoWjoyPc3d3Rs2dPXLlypRhfQcmaMkX6+vXXwCtjzYmIiPTH3Bz4/wvBNMKr8vHSpVI7PVu1ahXkcjmaNWuG7du349q1a7h06RKWL1+u8Wd1JWUonT17Nq5du4Y//vgDi18ZMzh+/Hjs27cPcXFxiImJwaFDh1SBrGrVqpDJZNi9ezcePnyIp0+fwtHRER9//DEmTJiADRs24MaNG4iJicGKFSuwYcMGnV5bSkoKEhMTcf/+fUREROCzzz5DrVq1VHVMnjwZ69evx+rVq3Ht2jV89dVX2LFjBz7++GMAQIcOHVC/fn0MHDgQMTExOHnyJAYPHoyQkBDVMAwfHx/ExcUhNjYWycnJyMzMxPr16/H999/j/PnzuHnzJn766SfY2tqiatWqqmMiIyNx7949rWE2N19fX2zcuBGXLl3CiRMnMHDgQNja2mq027p1K3744QdcvXoVs2bNwsmTJzFmzBid3rciK9TI42K2efNmYWVlJX744Qdx4cIFMWLECOHi4iKSkpK0tj958qT4+OOPxaZNm4SHh4fGYG0hhAgLCxPr1q0T58+fF7GxsaJr166iSpUq4mkBB98LYZwXwSnJ5ULUri1dT7B4saGrISIiY5XfxUOFsn27EF5e6he0eXtL24vR/fv3xejRo0XVqlWFlZWVqFy5snjzzTfFoUOHVG2Q6yI4IYQ4evSoqF+/vrCxsRGtW7cWW7duVbsIbsyYMaJGjRrC2tpauLm5iUGDBonk5GTV8Z999pnw8PAQMplMDBkyRAghhEKhEEuXLhW1a9cWlpaWws3NTYSFhYmIiAghhPaL5/ICQHWTyWSiUqVKIjw8XNy4cUOt3ddffy2qV68uLC0tRa1atcSPP/6otv/27dvizTffFPb29sLR0VH07dtXJCYmqva/ePFC9O7dW7i4uAgAYt26dWLnzp0iKChIODk5CXt7e9G8eXPx999/q46JiooSAQEBqovzhJAugnN2dtZ4HTExMaJJkybCxsZG+Pr6iq1bt2pcRAdArFq1SnTs2FFYW1sLHx8fsWXLlte+R6/S10Vwsv8vyigEBQWhadOmWLlyJQBpag9vb2989NFHmDZtWr7H+vj4YPz48Rg/fny+7R4+fAh3d3dERESgTZs2BaorLS0Nzs7OSE1NhZOTU4GOKUnffw+89570l6mbNwFLS0NXRERExubFixeIi4tDtWrVYGNjU7STyeXAkSNAQoI0N2fr1sXS80tlh0wmw86dO9GzZ88inSe/7+PC5DWjGQKRlZWFM2fOoEOHDqptZmZm6NChA6KiovT2PKmpqQCkMUV5yczMRFpamtrNmL3zDuDhIc1LvnmzoashIqIyz9wcaNsWGDBA+srwS6WM0QTg5ORkyOVyVKxYUW17xYoVkainKQ4UCgXGjx+Pli1bol69enm2mzt3LpydnVU3fQ1sLy7W1tLc4wCwYEGB5xYnIiIiMklGE4BLwujRo3H+/Hlsfk036fTp05Gamqq65b6y0lh9+CHg4ACcPw/s3WvoaoiIiIhyCCGKPPxBn4wmALu6usLc3Fy1drVSUlISPDw8inz+MWPGYPfu3Th06BC8vLzybWttbQ0nJye1m7FzcQE++EC6v2CBQUshIiIiMmpGE4CtrKzQuHFjHDhwQLVNoVDgwIEDeU5xUhBCCIwZMwY7d+7EwYMHUa1aNX2Ua5TGjwcsLIDDh4FTpwxdDREREZFxMpoADAATJ07Et99+iw0bNuDSpUsYOXIkMjIyMGzYMADA4MGDMX36dFX7rKws1SosWVlZuHfvHmJjY9VWTxk9ejR++ukn/Pzzz3B0dERiYiISExNVE0yXJV5ewNtvS/cXLjRsLUREZJyMaPInokLT1/evUU2DBgArV67EwoULkZiYiMDAQCxfvhxBQUEAgLZt28LHxwfr168HANy6dUtrj25ISAgOHz4MABrL9SmtW7cOQ4cOLVBNxj4NWm7nzgEBAYCZGXDlClCzpqErIiIiYyCXy3H16lW4u7ujQoUKhi6HSCePHj3CgwcPUKtWLdWqeUqFyWtGF4CNUWkKwADQrZu0FPvIkdIKcURERIC0LHBKSgrc3d1hZ2eXZycRkbERQuDZs2d48OABXFxcUKlSJY02DMB6VtoCcESENC2jjQ1w+zbg7m7oioiIyBgIIZCYmIiUlBRDl0KkExcXF3h4eGj95a0wec2iuAokw2nTBmjWDDh5Eli1Cpgzx9AVERGRMZDJZKhUqRLc3d3x8uVLQ5dDVCiWlpYawx50xR7gAihtPcAAsG0b0LcvUL48EB8P2NsbuiIiIiKi4lMql0Im/XrrLaBGDeDxY+CHHwxdDREREZHxYAAuo8zNgY8/lu4vXgxkZxu2HiIiIiJjwQBchg0ZAri5SRfCbd1q6GqIiIiIjAMDcBlmawt89JF0f+FCgKO9iYiIiBiAy7xRowA7O+Cff4Bcq0wTERERmSwG4DKuQgXgvfek+wsWGLYWIiIiImPAAGwCJkyQLorbv1/qCSYiIiIyZQzAJsDHBwgPl+4vXGjQUoiIiIgMjgHYREyeLH395Rfg1i2DlkJERERkUAzAJiIwEOjYEZDLgSVLDF0NERERkeEwAJuQKVOkr999Bzx6ZNhaiIiIiAyFAdiEtG8PNGwIPHsGfP21oashIiIiMgwGYBMik+X0Aq9YATx/bth6iIiIiAyBAdjE9OkjzQrx8CGwYYOhqyEiIiIqeQzAJsbCApg4Ubq/aJF0URwRERGRKWEANkHDhwPlywM3bgA7dxq6GiIiIqKSxQBsguztgTFjpPsLFgBCGLYeIiIiopLEAGyixowBbGyAU6eAyEhDV0NERERUchiATZSbGzBsmHR/wQLD1kJERERUkhiATdjEiYCZGbBnD3DunKGrISIiIioZDMAmrGZNoHdv6f6iRYathYiIiKikMACbuMmTpa8//wzcuWPYWoiIiIhKAgOwiWvaFGjbFsjOBpYtM3Q1RERERMWPAZhUyyOvXQukpBi0FCIiIqJixwBM6NwZqFcPePoUWLPG0NUQERERFS8GYIJMltMLvGwZ8OKFYeshIiIiKk4MwAQA6N8f8PYGEhOBn34ydDVERERExYcBmAAAlpbA+PHS/UWLAIXCoOUQERERFRsGYFIZMQJwdgauXAF27TJ0NURERETFgwGYVBwdgVGjpPtcHpmIiIjKKgZgUjN2LGBlBRw/Dhw7ZuhqiIiIiPSPAZjUeHgAQ4ZI9xcuNGwtRERERMWBAZg0TJokTY3222/A5cuGroaIiIhIvxiASUPt2kCPHtL9RYsMWwsRERGRvjEAk1bKhTE2bgTu3zdsLURERET6xABMWgUHA61aAVlZwPLlhq6GiIiISH8YgClPkydLX9esAdLSDFsLERERkb4wAFOeuncH6tQBUlOBb781dDVERERE+sEATHkyM8vpBV6yRBoOQURERFTaMQBTvgYOBCpVAu7dAzZtMnQ1REREREXHAEz5srYGxo+X7i9cCAhh0HKIiIiIiowBmF7rgw8AR0fgwgXgzz8NXQ0RERFR0TAA02s5O0shGAAWLDBsLURERERFxQBMBTJuHGBpCUREACdOGLoaIiIiIt0ZXQBetWoVfHx8YGNjg6CgIJw8eTLPthcuXEDv3r3h4+MDmUyGpUuXFvmcpJ2Xl3RBHCCNBSYiIiIqrYwqAG/ZsgUTJ07ErFmzEBMTgwYNGiAsLAwPHjzQ2v7Zs2eoXr065s2bBw8PD72ck/L28cfS1x07gGvXDFsLERERka6MKgB/9dVXGDFiBIYNGwY/Pz+sWbMGdnZ2+OGHH7S2b9q0KRYuXIj+/fvD2tpaL+ekvPn7A926STNBfPWVoashIiIi0o3RBOCsrCycOXMGHTp0UG0zMzNDhw4dEBUVZTTnNHVTpkhf160DkpIMWwsRERGRLowmACcnJ0Mul6NixYpq2ytWrIjExMQSPWdmZibS0tLUbiRp3RoICgIyM4GVKw1dDREREVHhGU0ANiZz586Fs7Oz6ubt7W3okoyGTJbTC7xqFfD0qWHrISIiIiosownArq6uMDc3R9Irf1dPSkrK8wK34jrn9OnTkZqaqrrduXNHp+cvq3r0AHx9gSdPgO+/N3Q1RERERIVjNAHYysoKjRs3xoEDB1TbFAoFDhw4gODg4BI9p7W1NZycnNRulMPcHJg0Sbr/1VfAy5eGrYeIiIioMIwmAAPAxIkT8e2332LDhg24dOkSRo4ciYyMDAwbNgwAMHjwYEyfPl3VPisrC7GxsYiNjUVWVhbu3buH2NhYXL9+vcDnJN0MHgy4uwPx8cDWrYauhoiIiKjgLAxdQG7h4eF4+PAhZs6cicTERAQGBmLv3r2qi9ji4+NhZpaT2e/fv4+GDRuqHi9atAiLFi1CSEgIDh8+XKBzkm5sbYGxY4FPP5WWRx4wQBofTERERGTsZEIIYegijF1aWhqcnZ2RmprK4RC5PH4MVKkCZGQA+/YBnToZuiIiIiIyVYXJa0Y1BIJKl/Llgffek+5zeWQiIiIqLRiAqUgmTJAuivv7byAmxtDVEBEREb0eAzAVSdWqQP/+0n32AhMREVFpwABMRTZ5svT1l1+AuDjD1kJERET0OgzAVGQNGgBhYYBCIc0LTERERGTMGIBJL5S9wN9/DyQnG7YWIiIiovwwAJNetGsHNGoEPH8OfP21oashIiIiyhsDMOmFTAZMmSLdX7ECePbMsPUQERER5YUBmPSmd2+gWjVpCMT69YauhoiIiEg7BmDSGwsLYNIk6f7ixYBcbth6iIiIiLRhACa9GjYMqFABuHkT2LHD0NUQERERaWIAJr2yswPGjJHuz58PCGHYeoiIiIhexQBMejd6NGBrC5w5Axw+bOhqiIiIiNQxAJPeubkBw4dL9xcsMGwtRERERK9iAKZiMXEiYGYG7N0L/PuvoashIiIiysEATMWienWgTx/p/qJFhq2FiIiIKDcGYCo2yuWRN20C4uMNWwsRERGREgMwFZsmTaQlkrOzgaVLDV0NERERkYQBmIqVcnnkb74BnjwxbC1EREREAAMwFbNOnYCAACAjA1izxtDVEBERETEAUzGTyXLGAi9bBrx4Ydh6iIiIiBiAqdiFhwPe3kBSErBxo6GrISIiIlPHAEzFztJSmhcYkKZEk8sNWw8RERGZNgZgKhHvvQeUKwdcvQr8/ruhqyEiIiJTxgBMJcLBARg5Urq/YAEghGHrISIiItOlcwCOj4/Hhx9+iNq1a6N8+fKIjIwEACQnJ2Ps2LH4559/9FYklQ0ffQRYWwPR0cCxY4auhoiIiEyVTgH44sWLaNiwIbZs2YJq1aohNTUV2dnZAABXV1ccPXoUK1eu1GuhVPp5eABDhkj3FywwbC1ERERkunQKwFOmTIGLiwuuXr2Kn376CeKVv2d369YNR44c0UuBVLZMmiRNjbZrF3DxoqGrISIiIlOkUwCOjIzEyJEj4ebmBplMprG/SpUquHfvXpGLo7KnVi3grbek+4sWGbYWIiIiMk06BWCFQgE7O7s89z98+BDW1tY6F0Vlm3JhjJ9+Au7fN2wtREREZHp0CsCNGjXCH3/8oXVfdnY2Nm/ejObNmxepMCq7mjcHWrcGXr6UVocjIiIiKkk6BeDp06dj7969GDlyJM6fPw8ASEpKwt9//41OnTrh0qVLmDZtml4LpbJlyhTp65o1QGqqYWshIiIi0yITr17BVkAbN27EuHHjkJqaCiEEZDIZhBBwcnLC6tWrMWDAAH3XajBpaWlwdnZGamoqnJycDF1OmaBQAPXrSxfCLViQMyyCiIiISBeFyWs6B2AAyMjIwP79+3Ht2jUoFArUqFEDYWFhcHR01PWURokBuHisWwcMHw54egJxcYCVlaErIiIiotKq2ANwZGQk6tatCzc3N637k5OTcfHiRbRp06awpzZKDMDFIzMTqF5duhBu3Tpg6FBDV0RERESlVWHymk5jgENDQ7F///489x84cAChoaG6nJpMiLU1MH68dH/hQmlYBBEREVFx0ykAv67TODMzE+bm5joVRKbl/fcBJydpLPCePYauhoiIiEyBRUEbxsfH49atW6rHly9fRmRkpEa7lJQUrF27FlWrVtVLgVS2OTsDH34oXQi3YAHQvbuhKyIiIqKyrsBjgOfMmYM5c+ZoXfktNyEEzM3NsXbtWgwfPlwvRRoaxwAXr/v3AR8faV7gqChpnmAiIiKiwihMXitwD3C/fv1Qr149CCHQr18/jB07Fq1bt1ZrI5PJYG9vj8DAQFSsWFG36snkeHoC77wjXQi3cCGwfbuhKyIiIqKyTKdZIDZs2ICQkBD4+PgUQ0nGhz3Axe/iRcDfH5DJgMuXgVq1DF0RERERlSbFPgvEkCFDTCb8Usnw8wPeeAMQAli82NDVEBERUVmm80IYL168wPbt2xETE4PU1FQoXpnDSiaT4fvvv9dLkYbGHuCScfQo0Lq1ND3a7dsAR9EQERFRQRXLGODcbt++jdDQUNy6dQsuLi5ITU1F+fLlkZKSArlcDldXVzg4OOhUPJmuli2lC+Cio4EVK4DPPzd0RURERFQW6TQEYvLkyUhNTUV0dDSuXr0KIQS2bNmCp0+fYv78+bC1tcW+ffv0XSuVcTIZMGWKdH/VKuDpU8PWQ0RERGWTTgH44MGDGDVqFJo1awYzM+kUQghYW1tj8uTJaN++PcYrl/giKoQ335QugEtJAb77ztDVEBERUVmkUwB+9uyZ6iI4JycnyGQypKamqvYHBwfj6NGjeimQTIu5OfDxx9L9r76S5gYmIiIi0iedAnCVKlVw9+5dAICFhQUqV66M6Oho1f6LFy/CxsZGPxWSyRk0SLoA7s4dYMsWQ1dDREREZY1OAbhdu3b47bffVI+HDh2KJUuWYMSIEXj33XexatUqvPHGGzoVtGrVKvj4+MDGxgZBQUE4efJkvu23bt2KOnXqwMbGBvXr18eePXvU9j99+hRjxoyBl5cXbG1t4efnhzVr1uhUG5UMGxtg7Fjp/sKF0tRoRERERHojdHD79m2xbds28eLFCyGEEM+fPxfvvvuucHFxERUqVBBDhgwRqamphT7v5s2bhZWVlfjhhx/EhQsXxIgRI4SLi4tISkrS2v7YsWPC3NxcLFiwQFy8eFF8+umnwtLSUpw7d07VZsSIEaJGjRri0KFDIi4uTqxdu1aYm5uL3377rcB1paamCgA6vSbSzePHQtjbCwEIsXevoashIiIiY1eYvKbzPMDFISgoCE2bNsXKlSsBAAqFAt7e3vjoo48wbdo0jfbh4eHIyMjA7t27VduaN2+OwMBAVS9vvXr1EB4ejhkzZqjaNG7cGF26dMHnBZxni/MAG8bEicCSJUC7dsCBA4auhoiIiIxZsa4E9+zZM1SoUAELFy7UuUBtsrKycObMGXTo0CGnODMzdOjQAVFRUVqPiYqKUmsPAGFhYWrtW7Rogd9//x337t2DEAKHDh3C1atX0alTpzxryczMRFpamtqNSt748YCFBXDwIHD6tKGrISIiorKi0AHYzs4OFhYWsLe312shycnJkMvlqPjK8l8VK1ZEYmKi1mMSExNf237FihXw8/ODl5cXrKys0LlzZ6xatQpt2rTJs5a5c+fC2dlZdfP29i7CKyNdVakC9O8v3dfz71tERERkwnS6CK53797Ytm0bjGj0RJ5WrFiB6Oho/P777zhz5gwWL16M0aNH4++//87zmOnTpyM1NVV1u3PnTglWTLlNnix93bYNuHnTsLUQERFR2aDTUsj9+/fHqFGjEBoaihEjRsDHxwe2trYa7Ro1alTgc7q6usLc3BxJSUlq25OSkuDh4aH1GA8Pj3zbP3/+HJ988gl27tyJbt26AQACAgIQGxuLRYsWaQyfULK2toa1tXWBa6fiExAAdO4M7N0rzQv8/8PDiYiIiHSmUwBu27at6v6RI0c09gshIJPJIJfLC3xOKysrNG7cGAcOHEDPnj0BSBfBHThwAGPGjNF6THBwMA4cOKC26tz+/fsRHBwMAHj58iVevnypWq1OydzcHAqFosC1kWFNmSIF4B9+AGbNAtzcDF0RERERlWY6BeB169bpuw4AwMSJEzFkyBA0adIEzZo1w9KlS5GRkYFhw4YBAAYPHozKlStj7ty5AIBx48YhJCQEixcvRrdu3bB582acPn0a33zzDQBplbqQkBBMnjwZtra2qFq1KiIiIvDjjz/iq6++KpbXQPrXti3QpIl0IdyqVcDs2YauiIiIiEozo5oGDQBWrlyJhQsXIjExEYGBgVi+fDmCgoIASD3PPj4+WL9+var91q1b8emnn+LWrVvw9fXFggUL0LVrV9X+xMRETJ8+HX/99RceP36MqlWr4v3338eECRMgk8kKVBOnQTO8X34BwsOBChWA+HjAzs7QFREREZExKUxeM7oAbIwYgA0vOxuoXVu6EG7lSmD0aENXRERERMakWOcBJjIECwtg0iTp/uLFUiAmIiIi0gUDMJUaQ4cCrq5AXBywfbuhqyEiIqLSigGYSg07O+Cjj6T7CxYAHLxDREREumAAplJl1CjA1haIiQEOHTJ0NURERFQaMQBTqeLqCrz7rnR/wQLD1kJERESlk86zQMjlcuzbtw83b97EkydPNJZFlslkmDFjhl6KNDTOAmFc4uKAmjUBhQKIjQUaNDB0RURERGRoxT4N2unTp9G7d2/cvXtXI/iqTlzIleCMGQOw8RkwANi8GRg4EPjpJ0NXQ0RERIZW7NOgjRo1Cs+fP8evv/6Kx48fQ6FQaNzKSvgl4zR5svR182bg9m3D1kJERESli04B+N9//8XUqVPxxhtvwMXFRc8lEb1eo0ZA+/aAXA4sXWroaoiIiKg00SkAe3l55Tn0gaikTJkiff32W+DxY8PWQkRERKWHTgF46tSp+Pbbb5GWlqbveogKrGNH6QK4jAxg9WpDV0NERESlhYUuB6Wnp8PBwQE1a9ZE//794e3tDXNzc7U2MpkMEyZM0EuRRNrIZFIv8MCBwPLlwMSJ0hzBRERERPnRaRYIM7PXdxxzFggqCS9fSlOixccDa9YAH3xg6IqIiIjIEAqT13TqAY6Li9OpMCJ9s7SUen7HjwcWLwbeew945Y8RRERERGp0XgjDlLAH2Lg9fQpUqQI8eQJs3w706mXoioiIiKikFXsPsFJGRgYiIiJw+/8nYq1atSpCQkJgb29flNMSFYqDAzB6NPD558D8+cBbb0njg4mIiIi00bkHeMWKFfj000/x9OlTtSnRHB0d8cUXX2DMmDF6K9LQ2ANs/B48kHqBMzOBiAigTRtDV0REREQlqdhXgvvxxx8xbtw41KtXDz///DNiY2MRGxuLTZs2oX79+hg3bhw2btyoU/FEunB3B4YOle4vWGDQUoiIiEyePEuO2KWHcfyjTYhdehjyLOOaGEGnHuDAwEC4uLjgwIEDGtOfyeVytG/fHikpKYiNjdVXnQbFHuDS4do1oHZtQAjg/HnA39/QFREREZme6Ck7UOWrcfCU31Vtu2/uhfiJy9B8QfFdqFPsPcBXrlxB3759NcIvAJibm6Nv3764cuWKLqcm0pmvb84FcIsWGbYWIiIiUxQ9ZQeaLewDj1zhFwA85PfQbGEfRE/ZYaDK1OkUgJ2dnXHr1q0899+6dYs9pWQQkydLX//3P+Du3fzbEhERkf7Is+So8tU4AEIjYJpBGnDg/dV4oxgOodMsEN26dcOKFSvQuHFj9O/fX23fli1bsHLlSgwcOFAvBRIVRlAQEBIiXQi3bBmwcKGhKyIiIir9FAogPR1IScm5paaqP5YfOII58rx7n8wgUFl+B7FfH0Hg+LYlUXaedBoD/PDhQ4SEhODKlSvw8PCAr68vAODatWtITExEnTp1EBERAVdXV70XbAgcA1y6/PEH0L074OgI3LkDODsbuiIiIiLDUiiAtDT1wJrX7dVgq9ymLTHa4yla4hja4jB6Yztq4dprazk+5me0WDFAPy8sl2KfB9jNzQ0xMTFYu3Yt/vzzT9U8wPXr18fUqVPx/vvvw8bGRpdTExVZly7SBXAXLgBr1wJTphi6IiIioqLJzs47wGoLrK/e0tL0U0d5y3R0sjuKULMItMg6jLrPTsNcFG5Ig12NSvoppgi4ElwBsAe49NmwQZoWrVIlIC4OsLY2dEVERGTKXr5UD6oFCa2526Wn66cOGxvAxSXvm7Oz+uMKlmnwuH4U5f49DLsTh2EWGwPIXwm81aoBISFQtGqDh+9/AjdFkmrMb24KyJBg7gWPZ3Ewt9KcSKGoSmwlOCJjNWAA8J//APfuSRfEDR9u6IqIiKg0y8oqWGjNq01Ghn7qsLPLP7DmF2ydnaUAnK+UFODoUeDwYemCmpgYafxEbtWrA23bShfdhIQAVasCkGZWiLviDLeFfaCATC0EKyAt0Xpn4lJULobwW1gF6gEODQ2FmZkZ9u3bBwsLC7Rr1+71J5bJcODAAb0UaWjsAS6dFi2SZoWoU0caDmGm05wnRERUFmRmFj605r49f66fOhwcChdaX31sZaWfOlSePAGOHJHC7uHDQGysZuCtWVMKusrQ6+2d7ym1zQN8z9wbdyYuNZp5gAvUAyyEgCLXm6FQKCCTyV57DJEhvf8+8N//ApcvSxfGvfGGoSsiIiJdCAG8eFH40Jq7zYsX+qnF0bHggfXVm5MTYGmpnzp09vixFHiVPbyxsZpXt9WqpR54K1cu1FM0X9AL8s97IPbrI3h2IwF2NSqh/qjWRtHzq8QxwAXAHuDSa9o0YP58oFUr6eediIhKnhBSD6quMxCkpEhDEPRBW69qQXtjnZwAi9I2ePTRIyAyMqeH999/NQNv7drqQxo8PQ1RaZEVJq/pFIAjIyNRt25duLm5ad2fnJyMixcvok2bNoU9tVFiAC697t+XxuZnZQHHjgEtWhi6IiKi0kcIaQyrrsMHUlKkWQyKSiYreGjV1sbREdCyiG3Z8vCheuA9d06zTd266j28Hh4lXWWxKPaL4EJDQ7Fx40a8/fbbWvcfOHAAb7/9NuSvXiVIVMI8PYFBg4Dvv5cWxdi509AVERGVPCHUFzEo7AwEKSmaF/7rwsys8KE1983BgddzaHjwQAq8yiEN589rtvHzk8Ju27ZAmzZAxYolXKTx0SkAv67TODMzE+Zl/lcsKi0mTZIC8G+/AVeuSH/pISIqTbStwlWY3tjUVM3rmnRhYVH40Jq7nYOD1ItLRZCUJAVdZQ/vxYuaberVy+ndbdMGcHcv6SqNXoEDcHx8PG7duqV6fPnyZURGRmq0S0lJwdq1a1H1/6fEIDK0unWBN98Efv8dWLwY+OYbQ1dERKZGLi/6Igb6uGLH0rJgYTWvYGtnxwBb4hIS1APv5cuabQICcoY0tGkDlJGVeItTgccAz5kzB3PmzCnQ7A/m5uZYu3YthpeRyVc5Brj0O3ZMuhDOygq4fbvMDHciohKSnZ0TVHUZPqCvVbisrIBy5XQfQmBjwwBr9O7fzwm7ERHSny5zk8mkwKsc0tC6NVChggEKNT7FMga4X79+qFevHoQQ6NevH8aOHYvWrVurtZHJZLC3t0dgYCAqcnwJGZGWLaUL4I4fB5YvB7780tAVEVFJenUVrsIOIXj6VD912NrqPnxAGWCpjLl7V72H99o19f0yGRAYmDOkoXVroHx5AxRatug0C8SGDRvQpk0bVKtWrThqMjrsAS4bfvsN6NlT+o/kzh3pamAiKh2ysoo2A8GzZ/qpw96+aIsYcFl2wp07OWH38GHgxg31/WZmQMOGOUMaWrWSuv3ptYp9FoiBAwfiWT7/mqSlpcHOzg4WpW6yPCrL3nhDugDuyhXg22+BiRMNXRGR6ci9iIEuPbH6XIVL11kInJ2NYBEDKn1u31Yf0nDzpvp+MzOgUaOcIQ2tWknfbFSsdOoBHjVqFCIjI3Fe21QbAOrXr4927dph2bJlRS7QGLAHuOz47jtgxAjAy0v6N4j/mRG9nrZVuArbG5uZqZ9anJx0H0Lg7FwKFzGg0ufWrZywe/iw9Dg3c3OgceOcIQ2tWknf2FRkxd4DvHfvXgwePDjP/X369MFPP/1UZgIwlR3vvAPMmCENudq8WZojmKisE0IaAqDrEILUVP2swiWT5QRRXWYgcHIygUUMqHQRAoiLUx/SEB+v3sbcHGjaNGdIQ8uWHINnBHQKwPfv30flfNaF9vT0xL1793Quiqi42NgA48YB06cDCxZIgZhXRJOxE0K6CEvXGQhSUvSzCpeZ2euHCOQXah0duYgBlXJCSH8+VIbdiAhpTG9uFhZS4FUOaWjRQhp7Q0ZFpwBcoUIFXHl1Wo5cLl26xKECZLQ+/BD44gtpsZy9e4EuXQxdEZV1CkVOgNVl+EBqqn5W4TI3L/oiBgywZFKEAK5fVx/S8GoHn6Ul0KxZzpCGFi2kqyXJqOkUgDt37oy1a9di4MCBaNiwodq+mJgYfPPNN+jbt69eCiTSNxcX4P33ga++kpZHZgCm11EotC9iUNDe2LQ0/a3CVa6c7kMI7O35Fw+ifAkBXL2qPqQhIUG9jaUl0Lx5zpCG4GBphRAqVXS6CO7+/fto2rQpHjx4gDfffBP+/v4AgPPnz2PXrl1wd3fHiRMn4OXlpfeCDYEXwZU9d+4A1atLfxY+eVL6axWVXXJ54RYxeLWNPlfhKsoiBra2DLBEeiWENDVQ7iENiYnqbayspMCr7OFt3pyB10gVJq/pFIABICEhAdOmTcNvv/2GtP9f4sbJyQk9e/bEl19+CU9PT11Oa5QYgMumIUOAH38E+vYFfvnF0NVQfnKvwqXLEIL0dP3UYWOj2xCC3IsYMMASGZAQwKVLOWE3IgJISlJvY20t9eoqA29QkPTbJxm9EgnASkIIPHz4EADg5ub22qWSSyMG4LLp3DlpNUkzM6kDoGZNQ1dUdmVlaQbUwlzMpa9VuOzsiraIAVfhIiplFArg4kX1eXj/P7Oo2NhI43aVQxqaNeMPeylV7NOg5SaTyWBtbQ0HB4cyGX6p7KpfXxr/++ef0njgr782dEXGKzOzaDMQ6HMVrqIsYmBlpZ86iMhIKRTAhQs5QxoiI4HkZPU2trZS4FX28DZrxiX6TJDOAfj06dP49NNPERkZiaysLPz1119o164dkpOT8e6772LChAlo27atHksl0r8pU6QA/MMPQKdO0mpTlSpJS62XpflGX7eIweuC7YsX+qnD0VH3IQRchYuINCgU0p/zcg9pePxYvY2dnTT3rjLwNm3K34ZJtwB8/PhxtGvXDpUrV8Y777yD7777TrXP1dUVqampWLt2rU4BeNWqVVi4cCESExPRoEEDrFixAs2aNcuz/datWzFjxgzcunULvr6+mD9/Prp27arW5tKlS5g6dSoiIiKQnZ0NPz8/bN++HVWqVCl0fVS2hIQANWpIS7G/9VbOdi8vYNkyoFcvw9WmJIQUzHWdgSA1VX+rcBV0EQNtwdbJiatwEVERyeXAv//mDGmIjASePFFvY28vra6mHNLQuDEDL2nQ6b+jTz75BHXr1kV0dDTS09PVAjAAhIaGYsOGDYU+75YtWzBx4kSsWbMGQUFBWLp0KcLCwnDlyhW4u7trtD9+/DgGDBiAuXPnonv37vj555/Rs2dPxMTEoF69egCAGzduoFWrVnj33XcxZ84cODk54cKFC7Dh+B4CsHOnFH5fde8e0KcPsG1b0UOwEEBGhu7DB1JSgJcvi1YDkLMKV1EWMShLveJEVArI5cDZszlDGo4ckf5RzM3BQQq8yh7exo355yJ6LZ0ugrO3t8fcuXMxduxYPHr0CG5ubvj777/Rrl07AMB3332HsWPH4lkhB/4FBQWhadOmWLlyJQBAoVDA29sbH330EaZNm6bRPjw8HBkZGdi9e7dqW/PmzREYGIg1a9YAAPr37w9LS0ts3LixsC9ThRfBlU1yOeDjIy2LrI1MJvUE37ypvQe2ML2x+ljEwMysaIsYcBUuIjJ62dlAbGzOkIYjR6R/YHNzdJTGqSl7eBs14p+XCEAJXARnaWkJRT6zut+7dw8OhVz2LysrC2fOnMH06dNV28zMzNChQwdERUVpPSYqKgoTJ05U2xYWFoZff/0VgBSg//jjD0yZMgVhYWH4559/UK1aNUyfPh09e/bMs5bMzExk5vqbsXKaNypbjhzJO/wCUs/tnTvSX870MQesuXnBFzHQFm4dHDiFFhGVMdnZQExMzpCGo0elibdzc3IC2rTJCbyBgQy8VGQ6fQc1b94c27Ztw/jx4zX2ZWRkYN26dQgJCSnUOZOTkyGXy1GxYkW17RUrVsTly5e1HpOYmKi1feL/T2L94MEDPH36FPPmzcPnn3+O+fPnY+/evejVqxcOHTqUZ41z587FnDlzClU/lT6vLu6TF2X4tbTUfQYCFxfpOgwGWCIyaS9fSoFX2cN79KjmRN3OzlLgVQ5pCAzk+CvSO50C8Jw5cxASEoJu3bphwIABAICzZ8/i5s2bWLRoER4+fIgZM2botVBdKHupe/TogQkTJgAAAgMDcfz4caxZsybPADx9+nS1nuW0tDR4e3sXf8FUoipVKli7rVuBrl25ChcRUaG9fAmcPq3ew5uRod6mXDn1Ht6AAAZeKnY6BeCgoCDs2bMHI0eOxODBgwEAkyZNAgDUqFEDe/bsQUBAQKHO6erqCnNzcyS9siJLUlISPDw8tB7j4eGRb3tXV1dYWFjAz89PrU3dunVx9OjRPGuxtraGNecELPNat5bG+N67p32Ig3IM8Ftv8d9iIqICycoCTp3KCbzHj2sG3vLlc3p427aVJmXnBQpUwnQeRNOuXTtcuXIFsbGxuHbtGhQKBWrUqIHGjRvrtCCGlZUVGjdujAMHDqjG5yoUChw4cABjxozRekxwcDAOHDigNhRj//79CA4OVp2zadOmuHLlitpxV69eRdWqVQtdI5Ut5ubSVGd9+khhN3cIVn4LL13K8EtElKfMTCnwKoc0HDsmXTWcW4UKOb27ISFAvXoMvGRwRR5FHhgYiMDAQD2UAkycOBFDhgxBkyZN0KxZMyxduhQZGRkYNmwYAGDw4MGoXLky5s6dCwAYN24cQkJCsHjxYnTr1g2bN2/G6dOn8c0336jOOXnyZISHh6NNmzYIDQ3F3r17sWvXLhw+fFgvNVPp1quXNNXZuHHqF8R5eUnh1xjmASYiMhqZmcCJE+o9vK+ulOPqmhN227YF/PwYeMnoFCgAR0ZGAgDatGmj9vi1J7ewgKurK2rVqlWg9uHh4Xj48CFmzpyJxMREBAYGYu/evaoL3eLj42GW64eoRYsW+Pnnn/Hpp5/ik08+ga+vL3799VfVHMAA8NZbb2HNmjWqadtq166N7du3o1WrVgWqicq+Xr2AHj2kWSESEsrmSnBERDp58QKIjs4JvNHRmoHX3T0n7LZtC9StywsmyOgVaB5gMzMzyGQyPH/+HFZWVqrHBeXt7Y3t27ejcePGRSrWUDgPMBERmYTnz6WQqxzSEB2tuZRkxYo5YTckBKhTh4GXjILe5wE+dOgQAGlMbe7HryOXy3H//n3MmzcPo0aNwokTJwp0HBEREZWAZ8+AqKicHt4TJ6QL2XKrVEl9SEOtWgy8VOoVKAC/Ol1YYef4ffbsGcaOHVuoY4iIiEjPMjKkcbvKwHvypOZa65Urqw9pqFmTgZfKnCJfBJeQkIAHDx6gZs2asLe319rmnXfeQVhYWFGfioiIiArj6VMp8CqHNJw8Ka2+lpuXl/qQhho1GHipzNM5AP/222+YOnUqrl27BkCafqxdu3ZITk5Gx44dMWvWLNV0ZnZ2dpx2jIiIqLilp0tTkSl7eE+f1gy8VaqoD2moVo2Bl0yOTgF4165d6NWrF4KDg/H2229j9uzZqn2urq6oXLky1q1bpwrAREREVAzS0qTAe/iwdDtzBpDL1dv4+KgPafDxKekqiYyOTgH4s88+Q5s2bXDo0CE8evRILQAD0gIVa9eu1Ud9REREpJSaKi0nrBzScOYMoFCot6lWTX1IA/8CS6RBpwB8/vx5fPXVV3nur1ixIh48eKBzUURERAQgJUWapFw5pOGffzQDb40aOWE3JEQa4kBE+dIpANvZ2SHj1bW9c7l58yYqVKigc1FEREQm6ckTKfAqhzTExqqv0w4Avr7qSwt7eZV8nUSlnE4BODQ0FBs2bMD48eM19iUmJuLbb79F9+7di1obERFR2fb4MRAZmTOk4exZzcBbq5Z6D2/lyoaolKhM0SkAf/HFF2jevDmaNm2Kvn37QiaTYd++fTh48CDWrl0LIQRmzZql71qJiIhKt+RkKfAqhzScO6cZeOvUUQ+8lSoZolKiMq1ASyFrc+HCBYwbNw6HDh1C7lO0bdsWq1atQt26dfVWpKFxKWQiItLJw4c5PbyHDwPnz2u28fPLGdLQpg3g4VHCRRKVDXpfClkbf39//P3333jy5AmuX78OhUKB6tWrw83NDQAghICM8woSEZEpefAgp3c3IgK4cEGzjb9/Tg9vmzZAxYolXSWRySvySnDlypVD06ZNVY+zsrKwfv16LFq0CFevXi3q6YmIiIxXYqIUdJWh99IlzTb166sH3v/vKCIiwylUAM7KysLvv/+OGzduoFy5cujevTs8PT0BAM+ePcPKlSuxdOlSJCYmokaNGsVSMBERkcEkJKj38F6+rNmmQYOcIQ2tWwOuriVdJRG9RoED8P3799G2bVvcuHFDNebX1tYWv//+O6ysrPD222/j3r17aNasGVasWIFevXoVW9FEREQl4t499cD76l82ZTIp8Cp7eFu3BjgNKJHRK3AA/s9//oO4uDhMmTIFrVu3RlxcHD777DO8//77SE5Ohr+/P3766SeEhIQUZ71ERETF5+7dnLB7+DBw/br6fpkMaNhQPfCWK2eAQomoKAocgPfv349hw4Zh7ty5qm0eHh7o27cvunXrht9++w1mZmbFUiQREVGxiI9X7+G9cUN9v5kZ0KhRzpCGVq0AFxcDFEpE+lTgAJyUlITmzZurbVM+Hj58OMMvEREZv9u3c6Yki4gA4uLU95uZAY0b5/TwtmoFODsboFAiKk4FDsByuRw2NjZq25SPnfmPAxERGRshgFu31Ic03L6t3sbcHGjSJKeHt2VLgPO9E5V5hZoF4tatW4iJiVE9Tk1NBQBcu3YNLlr+JNSoUaOiVUdERFRQQgA3b6oPaYiPV29jYSEF3rZtpVuLFoCjowGKJSJDKvBKcGZmZloXttC24IVym1wu10+VBsaV4IiIjJAQ0pjd3D28d++qt7GwAJo1yxnS0KIF4OBggGKJqLgVy0pw69atK3JhREREOhMCuHZNPfDev6/extISCArKGdIQHAzY2xugWCIyZgUOwEOGDCnOOoiIiNQJAVy5oj6kISFBvY2VlRR4lUMamjcH7OwMUCwRlSZFXgqZiIhIL4SQVlbL3cOblKTextpaCrnKIQ3NmwO2tgYolohKMwZgIiIyDCGAixfVe3gfPFBvY2MjDWNQDmkICpK2EREVAQMwERGVDIVCCrzKeXgjI4GHD9Xb2NhIF6ophzQ0ayb1+hIR6REDMBERFQ+FAjh/Pqd3NyICePRIvY2trTT3rnJIQ9OmDLxEVOwYgImISD8UCuDff3OGNERGAo8fq7exs5NWV1MOaWjSRLqQjYioBDEAExGRbuRyKfAqhzQcOQI8eaLext5eCrzKHt4mTaSpyoiIDIgBmIiICkYuB2Jjc4Y0REYC/78iqIqDA9C6dU7gbdSIgZeIjA4DMBERaZedDfzzT86QhiNHgLQ09TZOTlLgVQ5paNhQWn2NiMiI8V8pIiKSZGcDMTE5QxqOHgXS09XbODur9/AGBjLwElGpw3+1iIhM1cuXwJkzOUMajh4Fnj5Vb+PiArRpkxN4GzQAzM0NUCwRkf4wABMRmYqsLOD06ZwhDceOARkZ6m3KlZOCrnJIQ/36DLxEVOYwABMRlVVZWcCpUzlDGo4fB549U29ToYJ6D2/9+oCZmQGKJSIqOQzARERlRWYmcPJkzpCG48eB58/V27i65vTuhoQA/v4MvERkchiAiYhKqxcvgBMncoY0REVJ23Jzc8sJu23bAnXrMvASkcljACYiKi1evACio3OGNERHS72+uVWsqN7DW7cuIJMZoFgiIuPFAExEZKyeP5d6dZVDGqKjpXG9uXl4qPfw1q7NwEtE9BoMwERExuLZM2ncrnJIw8mTmoHX01M98Pr6MvASERUSAzARkaFkZEiBV9nDe/KkNDdvbpUrS0FXGXpr1mTgJSIqIgZgIqKS8vSpNPeuMvCeOiWtvpabt7d6D2/16gy8RER6xgBMRFRc0tOl1dWUQxpOnwbkcvU2VaoAoaE5gdfHh4GXiKiYMQATEelLWpoUeJU9vGfOaAbeatXUZ2nw8TFAoUREpo0BmIhIVykp6oE3JgZQKNTbVK+eE3ZDQoCqVQ1QKBER5cYATERUUCkpwJEjOfPwxsZqBt6aNdV7eL29S7xMIiLKHwMwEVFeHj/OCbwREVLgFUK9Ta1a6oG3cmUDFEpERIXBAExEpPToERAZmXPR2r//agbe2rXVhzR4ehqiUiIiKgKjXBB+1apV8PHxgY2NDYKCgnDy5Ml822/duhV16tSBjY0N6tevjz179uTZ9sMPP4RMJsPSpUv1XDURlTrJycCOHcDYsUBAAODqCvTqBSxbBpw9K4XfunWBDz8ENm8G7t8HLl8G1qwBBgxg+CUiKqWMrgd4y5YtmDhxItasWYOgoCAsXboUYWFhuHLlCtzd3TXaHz9+HAMGDMDcuXPRvXt3/Pzzz+jZsydiYmJQr149tbY7d+5EdHQ0PPmfFpFpevBA6uFVDmk4f16zjZ9fzsITbdoAFSuWcJFERFTcZEK8+vc9wwoKCkLTpk2xcuVKAIBCoYC3tzc++ugjTJs2TaN9eHg4MjIysHv3btW25s2bIzAwEGvWrFFtu3fvHoKCgrBv3z5069YN48ePx/jx4wtUU1paGpydnZGamgonJ6eivUAiKjlJSVLQVQ5puHhRs029ejlDGtq0AbT8ok1ERMavMHnNqHqAs7KycObMGUyfPl21zczMDB06dEBUVJTWY6KiojBx4kS1bWFhYfj1119VjxUKBQYNGoTJkyfD39//tXVkZmYiMzNT9TgtLa2Qr4SIDCIxMSfsHj4sDVd4VUBAzkVrbdpIwx6IiMikGFUATk5OhlwuR8VX/uRYsWJFXNb2HxmAxMREre0TExNVj+fPnw8LCwuMHTu2QHXMnTsXc+bMKWT1RFTi7t/PCbwREcCVK+r7ZTIp8CqHNLRuDVSoYIBCiYjImBhVAC4OZ86cwbJlyxATEwNZAZcXnT59ulqvclpaGrw5lyeR4d29qz6k4do19f0yGRAYmDOkoXVroHx5AxRKRETGzKgCsKurK8zNzZGUlKS2PSkpCR4eHlqP8fDwyLf9kSNH8ODBA1SpUkW1Xy6XY9KkSVi6dClu3bqlcU5ra2tYW1sX8dUQUZHduaM+pOHGDfX9ZmZAw4Y5QxpatQLKlTNAoUREVJoYVQC2srJC48aNceDAAfTs2ROANH73wIEDGDNmjNZjgoODceDAAbUL2vbv34/g4GAAwKBBg9ChQwe1Y8LCwjBo0CAMGzasWF4HEeno9m31IQ03b6rvNzMDGjXKGdLQqhXg7GyAQomIqDQzqgAMABMnTsSQIUPQpEkTNGvWDEuXLkVGRoYqrA4ePBiVK1fG3LlzAQDjxo1DSEgIFi9ejG7dumHz5s04ffo0vvnmGwBAhQoVUOGVMX+Wlpbw8PBA7dq1S/bFEZG6W7dywu7hw9Lj3MzNgcaNc4Y0tGoFcCYWIiIqIqMLwOHh4Xj48CFmzpyJxMREBAYGYu/evaoL3eLj42FmlrN+R4sWLfDzzz/j008/xSeffAJfX1/8+uuvGnMAE5GBCQHExan38N6+rd7G3Bxo2jRnSEPLloCjoyGqJSKiMszo5gE2RpwHmEgHQkhDGJTjdyMipDG9uVlYSIFXOaShRQvAwaHkayUiolKv1M4DTESlmBDA9evqQxru3VNvY2kJNGuWM6ShRQvA3t4AxRIRkSljACYi3QgBXL2qPqTh/n31NpaWQPPmOUMagoMBOztDVEtERKTCAExEBSOEtNBE7iENuRacAQBYWUmBV9nD27w5Ay8RERkdBmAi0k4I4NKlnLAbEQG8Muc2rK2lXl1lD29QEGBra4hqiYiICowBmIgkCgVw8aL6kIaHD9Xb2NhI43aVgbdZM2kbERFRKcIATGSqFArgwoWcIQ2RkUBysnobW1sp8CqHNDRrJvX6EhERlWIMwESmQqEAzp1TH9Lw+LF6Gzs7ae5dZQ9v06bSuF4iIqIyhAGYqKySy4F//80Z0hAZCTx5ot7G3l5aXU0ZeBs3ZuAlIqIyjwGYqKyQy4GzZ3OGNBw5AqSkqLdxcJACr3JIQ+PG0lRlREREJoQBmKi0ys4GYmNzhjQcOQKkpqq3cXQEWrfO6eFt1EhafY2IiMiE8X9CotIiOxuIickZ0nD0KJCWpt7GyQlo0yYn8AYGMvASERG9gv8zEhmrly+lwKvs4T16FEhPV2/j7CwFXuWQhsBAwNzcAMUSERGVHgzARMbi5Uvg9Gn1Ht6MDPU25cqp9/AGBDDwEhERFRIDMJGhZGUBp07lBN7jxzUDb/nyOT28bdsC9esDZmYGKJaIiKjsYAAmKimZmVLgVQ5pOHYMeP5cvU2FCjm9uyEhQL16DLxERER6xgBMVFwyM4ETJ3J6eKOiNAOvq2tO2G3bFvDzY+AlIiIqZgzARPry4gUQHZ0TeKOjpW25ubvnhN22bYG6dQGZzADFEhERmS4GYCJAWkTiyBEgIQGoVEmaO/d1F5c9fy6FXOWQhuhoqdc3t4oVc8JuSAhQpw4DLxERkYExABPt2AGMGwfcvZuzzcsLWLYM6NUrZ9uzZ9IwBmUP74kT0oVsuVWqpD6koVYtBl4iIiIjwwBMpm3HDqBPH0AI9e337knbZ8yQeocPHwZOnpSmKsvN0zOnh7dtW6BmTQZeIiIiIycT4tX/+elVaWlpcHZ2RmpqKpycnAxdDumLXA74+Kj3/L6Ol5d6D2+NGgy8RERERqAweY09wGS6jhwpWPjt1AkID5cCb7VqDLxERESlHAMwma6EhIK1GzoUGDCgWEshIiKiksMJR8k0vXwpXcxWEJUqFW8tREREVKLYA0ymJyoK+OAD4Ny5/NvJZNKY39atS6YuIiIiKhHsASbT8eQJ8OGHQMuWUvitUAEYPVoKuq+O61U+Xrr09fMBExERUanCAExlnxDAzz9Li1CsXSs9HjoUuHwZWLkS2LYNqFxZ/RgvL2l77nmAiYiIqEzgEAgq265fB0aNAvbvlx7XqQOsWSNNY6bUqxfQo0fhV4IjIiKiUokBmMqmzExgwQLgiy+k+9bWwKefApMnS/dfZW4uTXNGREREZR4DMJU9hw9LY32vXJEed+wIfP21tEobERERmTyOAaayIzlZGtsbGiqF34oVpbG/+/Yx/BIREZEKAzCVfgoF8MMPQO3awIYN0gwOI0dKF7kNGMCV24iIiEgNh0BQ6XbxojTc4cgR6XFAgDTTQ/Pmhq2LiIiIjBZ7gKl0ev4c+M9/gMBAKfza2QELFwKnTzP8EhERUb7YA0ylz9690gIWN29Kj994A1ixAqha1bB1ERERUanAHmAqPRISgPBwoEsXKfx6eQE7dwK//cbwS0RERAXGAEzGTy4HVq2SFrH45RfAzAyYMEEa/9uzJy9yIyIiokLhEAgybrGxwAcfACdPSo+bNpUucmvY0KBlERERUenFHmAyTk+fAhMnAo0bS+HX0RFYuRKIimL4JSIioiJhDzAZn19/BT76CLh7V3rcrx+wZAng6WnQsoiIiKhsYAAm4xEfLwXf33+XHlerJi1h3LmzYesiIiKiMoVDIMjwsrOBxYsBPz8p/FpYANOnA+fPM/wSERGR3rEHmAzrxAnpIrezZ6XHrVoBa9YA/v6GrYuIiIjKLPYAk2GkpACjRgHBwVL4LV8e+O47ICKC4ZeIiIiKFXuAqWQJAWzZIs3jm5gobRs8GFi0CHBzM2xtREREZBIYgKnk3Lgh9fr+9Zf0uFYtabhDaKhh6yIiIiKTwiEQVPyysoAvvgDq1ZPCr7U1MGcO8O+/DL9ERERU4owyAK9atQo+Pj6wsbFBUFAQTipXAcvD1q1bUadOHdjY2KB+/frYs2ePat/Lly8xdepU1K9fH/b29vD09MTgwYNx//794n4ZBACRkUBgIPDpp8CLF0D79sC5c8DMmVIQJiIiIiphRheAt2zZgokTJ2LWrFmIiYlBgwYNEBYWhgcPHmhtf/z4cQwYMADvvvsu/vnnH/Ts2RM9e/bE+fPnAQDPnj1DTEwMZsyYgZiYGOzYsQNXrlzBm2++WZIvy/QkJwPDhwMhIcClS4C7O/DTT8D+/YCvr6GrIyIiIhMmE0IIQxeRW1BQEJo2bYqVK1cCABQKBby9vfHRRx9h2rRpGu3Dw8ORkZGB3bt3q7Y1b94cgYGBWLNmjdbnOHXqFJo1a4bbt2+jSpUqr60pLS0Nzs7OSE1NhZOTk46vzEQIAWzYAHz8MfDokbTt/feBefOAcuUMWxsRERGVWYXJa0bVA5yVlYUzZ86gQ4cOqm1mZmbo0KEDoqKitB4TFRWl1h4AwsLC8mwPAKmpqZDJZHBxcdG6PzMzE2lpaWo3KoBLl6QxvcOGSeG3Xj3g2DFg7VqGXyIiIjIaRhWAk5OTIZfLUbFiRbXtFStWRKJyyqxXJCYmFqr9ixcvMHXqVAwYMCDP3w7mzp0LZ2dn1c3b21uHV2NCnj8HZswAGjSQ5vG1tQXmzwdiYoAWLQxdHREREZEaowrAxe3ly5fo168fhBBYvXp1nu2mT5+O1NRU1e3OnTslWGUps38/UL8+8PnnwMuXQLduwMWLwJQpgKWloasjIiIi0mBU8wC7urrC3NwcSUlJatuTkpLg4eGh9RgPD48CtVeG39u3b+PgwYP5jg2xtraGNWcoyF9iIjBxIrBpk/TY0xNYvhzo1QuQyQxbGxEREVE+jKoH2MrKCo0bN8aBAwdU2xQKBQ4cOIDg4GCtxwQHB6u1B4D9+/ertVeG32vXruHvv/9GhQoViucFmAKFQlq8ok4dKfyamQFjx0rjf3v3ZvglIiIio2dUPcAAMHHiRAwZMgRNmjRBs2bNsHTpUmRkZGDYsGEAgMGDB6Ny5cqYO3cuAGDcuHEICQnB4sWL0a1bN2zevBmnT5/GN998A0AKv3369EFMTAx2794NuVyuGh9cvnx5WFlZGeaFlkZnzwIffACcOCE9btxYusCtcWPD1kVERERUCEYXgMPDw/Hw4UPMnDkTiYmJCAwMxN69e1UXusXHx8PMLKfjukWLFvj555/x6aef4pNPPoGvry9+/fVX1KtXDwBw7949/P777wCAwMBAtec6dOgQ2rZtWyKvq1R7+hSYPRtYuhSQywFHR2llt1GjAHNzQ1dHREREVChGNw+wMTLpeYB37QLGjAHi46XHffpIQbhyZYOWRURERJRbYfKa0fUAk5G4cwcYNw7YuVN6XLUqsGqVNMsDERERUSlmVBfBkRHIzgaWLAH8/KTwa2EBTJ0KXLjA8EtERERlAnuAKcepU9JFbv/8Iz1u0UKa8aF+fcPWRURERKRH7AEmIDVVGucbFCSFXxcX4JtvgCNHGH6JiIiozGEPsCkTAti6FRg/HkhIkLa98w6weDHg7m7Q0oiIiIiKCwOwqbp5Exg9Gti7V3rs6wusXg20b2/YuoiIiIiKGYdAmJqsLGDuXMDfXwq/VlbArFnAv/8y/BIREZFJYA+wKTl6FPjwQ2lGBwAIDZV6fWvXNmxdRERERCWIPcCm4NEj4L33gNatpfDr6gr8+CNw4ADDLxEREZkc9gCXZUIAGzcCkyYBycnStvfeA+bPB8qXN2xtRERERAbCAFxWXbkCjBwJHDokPfb3l+b0bdXKsHURERERGRiHQJQ1L15IF7UFBEjh19ZWuugtJobhl4iIiAjsAS5b/v5b6vW9fl163KULsGoVUK2aYesiIiIiMiLsAS4LkpKkBSw6dpTCb6VKwC+/AH/8wfBLRERE9AoG4NJMoZCWLK5TB/jf/wCZTFrS+NIloG9f6TERERERqeEQiNLq3Dnggw+AqCjpccOGwNq1QNOmhq2LiIiIyMixB7i0ycgApkyRAm9UFODgACxZApw8yfBLREREVADsATY2cjlw5AiQkCCN5W3dGjA3l/bt3i0Ncbh9W3rcqxewbBng5WW4eomIiIhKGQZgY7JjBzBuHHD3bs42Ly9g5kxg3z5g+3ZpW5Uq0uwO3bsbpk4iIiKiUowB2Fjs2AH06SOt3pbb3bvA++9L983NgQkTgNmzAXv7Ei+RiIiIqCxgADYGcrnU8/tq+M3Nykoa89uoUcnVRURERFQG8SI4Y3DkiPqwB22ysoC0tJKph4iIiKgMYwA2BgkJ+m1HRERERHliADYGlSrptx0RERER5YkB2Bi0bi3N9pDXym0yGeDtLbUjIiIioiJhADYG5ubSfL6AZghWPl66NGc+YCIiIiLSGQOwsejVC9i2DahcWX27l5e0vVcvw9RFREREVMZwGjRj0qsX0KNH3ivBEREREVGRMQAbG3NzoG1bQ1dBREREVGZxCAQRERERmRQGYCIiIiIyKQzARERERGRSGICJiIiIyKQwABMRERGRSWEAJiIiIiKTwgBMRERERCaFAZiIiIiITAoDMBERERGZFAZgIiIiIjIpXAq5AIQQAIC0tDQDV0JERERE2ihzmjK35YcBuADS09MBAN7e3gauhIiIiIjyk56eDmdn53zbyERBYrKJUygUuH//PhwdHSGTyQBIv2V4e3vjzp07cHJyMnCFpA/8TMsmfq5lDz/TsoefadlU0p+rEALp6enw9PSEmVn+o3zZA1wAZmZm8PLy0rrPycmJP6xlDD/Tsomfa9nDz7Ts4WdaNpXk5/q6nl8lXgRHRERERCaFAZiIiIiITAoDsI6sra0xa9YsWFtbG7oU0hN+pmUTP9eyh59p2cPPtGwy5s+VF8ERERERkUlhDzARERERmRQGYCIiIiIyKQzARERERGRSGICJiIiIyKQwAOtg1apV8PHxgY2NDYKCgnDy5ElDl0T5iIyMxBtvvAFPT0/IZDL8+uuvavuFEJg5cyYqVaoEW1tbdOjQAdeuXVNr8/jxYwwcOBBOTk5wcXHBu+++i6dPn5bgqyCluXPnomnTpnB0dIS7uzt69uyJK1euqLV58eIFRo8ejQoVKsDBwQG9e/dGUlKSWpv4+Hh069YNdnZ2cHd3x+TJk5GdnV2SL4VyWb16NQICAlQT5gcHB+PPP/9U7ednWvrNmzcPMpkM48ePV23j51r6zJ49GzKZTO1Wp04d1f7S8pkyABfSli1bMHHiRMyaNQsxMTFo0KABwsLC8ODBA0OXRnnIyMhAgwYNsGrVKq37FyxYgOXLl2PNmjU4ceIE7O3tERYWhhcvXqjaDBw4EBcuXMD+/fuxe/duREZG4v333y+pl0C5REREYPTo0YiOjsb+/fvx8uVLdOrUCRkZGao2EyZMwK5du7B161ZERETg/v376NWrl2q/XC5Ht27dkJWVhePHj2PDhg1Yv349Zs6caYiXRAC8vLwwb948nDlzBqdPn0a7du3Qo0cPXLhwAQA/09Lu1KlTWLt2LQICAtS283Mtnfz9/ZGQkKC6HT16VLWv1HymggqlWbNmYvTo0arHcrlceHp6irlz5xqwKiooAGLnzp2qxwqFQnh4eIiFCxeqtqWkpAhra2uxadMmIYQQFy9eFADEqVOnVG3+/PNPIZPJxL1790qsdtLuwYMHAoCIiIgQQkifn6Wlpdi6dauqzaVLlwQAERUVJYQQYs+ePcLMzEwkJiaq2qxevVo4OTmJzMzMkn0BlKdy5cqJ7777jp9pKZeeni58fX3F/v37RUhIiBg3bpwQgj+rpdWsWbNEgwYNtO4rTZ8pe4ALISsrC2fOnEGHDh1U28zMzNChQwdERUUZsDLSVVxcHBITE9U+U2dnZwQFBak+06ioKLi4uKBJkyaqNh06dICZmRlOnDhR4jWTutTUVABA+fLlAQBnzpzBy5cv1T7TOnXqoEqVKmqfaf369VGxYkVVm7CwMKSlpal6HMlw5HI5Nm/ejIyMDAQHB/MzLeVGjx6Nbt26qX1+AH9WS7Nr167B09MT1atXx8CBAxEfHw+gdH2mFiX2TGVAcnIy5HK52ocGABUrVsTly5cNVBUVRWJiIgBo/UyV+xITE+Hu7q6238LCAuXLl1e1IcNQKBQYP348WrZsiXr16gGQPi8rKyu4uLiotX31M9X2mSv3kWGcO3cOwcHBePHiBRwcHLBz5074+fkhNjaWn2kptXnzZsTExODUqVMa+/izWjoFBQVh/fr1qF27NhISEjBnzhy0bt0a58+fL1WfKQMwEZVao0ePxvnz59XGn1HpVbt2bcTGxiI1NRXbtm3DkCFDEBERYeiySEd37tzBuHHjsH//ftjY2Bi6HNKTLl26qO4HBAQgKCgIVatWxS+//AJbW1sDVlY4HAJRCK6urjA3N9e4mjEpKQkeHh4GqoqKQvm55feZenh4aFzkmJ2djcePH/NzN6AxY8Zg9+7dOHToELy8vFTbPTw8kJWVhZSUFLX2r36m2j5z5T4yDCsrK9SsWRONGzfG3Llz0aBBAyxbtoyfaSl15swZPHjwAI0aNYKFhQUsLCwQERGB5cuXw8LCAhUrVuTnWga4uLigVq1auH79eqn6WWUALgQrKys0btwYBw4cUG1TKBQ4cOAAgoODDVgZ6apatWrw8PBQ+0zT0tJw4sQJ1WcaHByMlJQUnDlzRtXm4MGDUCgUCAoKKvGaTZ0QAmPGjMHOnTtx8OBBVKtWTW1/48aNYWlpqfaZXrlyBfHx8Wqf6blz59R+sdm/fz+cnJzg5+dXMi+EXkuhUCAzM5OfaSnVvn17nDt3DrGxsapbkyZNMHDgQNV9fq6l39OnT3Hjxg1UqlSpdP2sltjldmXE5s2bhbW1tVi/fr24ePGieP/994WLi4va1YxkXNLT08U///wj/vnnHwFAfPXVV+Kff/4Rt2/fFkIIMW/ePOHi4iJ+++038e+//4oePXqIatWqiefPn6vO0blzZ9GwYUNx4sQJcfToUeHr6ysGDBhgqJdk0kaOHCmcnZ3F4cOHRUJCgur27NkzVZsPP/xQVKlSRRw8eFCcPn1aBAcHi+DgYNX+7OxsUa9ePdGpUycRGxsr9u7dK9zc3MT06dMN8ZJICDFt2jQREREh4uLixL///iumTZsmZDKZ+Ouvv4QQ/EzLityzQAjBz7U0mjRpkjh8+LCIi4sTx44dEx06dBCurq7iwYMHQojS85kyAOtgxYoVokqVKsLKyko0a9ZMREdHG7okysehQ4cEAI3bkCFDhBDSVGgzZswQFStWFNbW1qJ9+/biypUraud49OiRGDBggHBwcBBOTk5i2LBhIj093QCvhrR9lgDEunXrVG2eP38uRo0aJcqVKyfs7OzEW2+9JRISEtTOc+vWLdGlSxdha2srXF1dxaRJk8TLly9L+NWQ0vDhw0XVqlWFlZWVcHNzE+3bt1eFXyH4mZYVrwZgfq6lT3h4uKhUqZKwsrISlStXFuHh4eL69euq/aXlM5UJIUTJ9TcTERERERkWxwATERERkUlhACYiIiIik8IATEREREQmhQGYiIiIiEwKAzARERERmRQGYCIiIiIyKQzARERERGRSGICJiIrR4cOHIZPJsG3bNkOXUiBJSUno06cPKlSoAJlMhqVLlxbpfD4+Phg6dKheaiMi0hcGYCIq9davXw+ZTAYbGxvcu3dPY3/btm1Rr149A1RW+kyYMAH79u3D9OnTsXHjRnTu3NnQJeXp2bNnmD17Ng4fPmzoUoiolLEwdAFERPqSmZmJefPmYcWKFYYupdQ6ePAgevTogY8//tjQpbzWs2fPMGfOHADSLzlERAXFHmAiKjMCAwPx7bff4v79+4YupcRlZGTo5TwPHjyAi4uLXs5VWunrvSQi48UATERlxieffAK5XI558+bl2+7WrVuQyWRYv369xj6ZTIbZs2erHs+ePRsymQxXr17FO++8A2dnZ7i5uWHGjBkQQuDOnTvo0aMHnJyc4OHhgcWLF2t9Trlcjk8++QQeHh6wt7fHm2++iTt37mi0O3HiBDp37gxnZ2fY2dkhJCQEx44dU2ujrOnixYt4++23Ua5cObRq1Srf13zz5k307dsX5cuXh52dHZo3b44//vhDtV85jEQIgVWrVkEmk0Emk+V7ToVCgWXLlqF+/fqwsbGBm5sbOnfujNOnT+d5jLL2Vymf/9atW6ptp0+fRlhYGFxdXWFra4tq1aph+PDhAKTP0M3NDQAwZ84cVb25P7vLly+jT58+KF++PGxsbNCkSRP8/vvvWp83IiICo0aNgru7O7y8vAAA6enpGD9+PHx8fGBtbQ13d3d07NgRMTEx+b4vRGT8OASCiMqMatWqYfDgwfj2228xbdo0eHp66u3c4eHhqFu3LubNm4c//vgDn3/+OcqXL4+1a9eiXbt2mD9/Pv73v//h448/RtOmTdGmTRu147/44gvIZDJMnToVDx48wNKlS9GhQwfExsbC1tYWgDT8oEuXLmjcuDFmzZoFMzMzrFu3Du3atcORI0fQrFkztXP27dsXvr6++PLLLyGEyLP2pKQktGjRAs+ePcPYsWNRoUIFbNiwAW+++Sa2bduGt956C23atMHGjRsxaNAgdOzYEYMHD37te/Luu+9i/fr16NKlC9577z1kZ2fjyJEjiI6ORpMmTXR4l3M8ePAAnTp1gpubG6ZNmwYXFxfcunULO3bsAAC4ublh9erVGDlyJN566y306tULABAQEAAAuHDhAlq2bInKlStj2rRpsLe3xy+//IKePXti+/bteOutt9Seb9SoUXBzc8PMmTNVPcAffvghtm3bhjFjxsDPzw+PHj3C0aNHcenSJTRq1KhIr4+IDEwQEZVy69atEwDEqVOnxI0bN4SFhYUYO3asan9ISIjw9/dXPY6LixMAxLp16zTOBUDMmjVL9XjWrFkCgHj//fdV27Kzs4WXl5eQyWRi3rx5qu1PnjwRtra2YsiQIapthw4dEgBE5cqVRVpammr7L7/8IgCIZcuWCSGEUCgUwtfXV4SFhQmFQqFq9+zZM1GtWjXRsWNHjZoGDBhQoPdn/PjxAoA4cuSIalt6erqoVq2a8PHxEXK5XO31jx49+rXnPHjwoACg9j4r5a6/atWqau+HsvZXKT/DuLg4IYQQO3fuVH2meXn48KHG56XUvn17Ub9+ffHixQu1ulq0aCF8fX01nrdVq1YiOztb7RzOzs4Fei+IqPThEAgiKlOqV6+OQYMG4ZtvvkFCQoLezvvee++p7pubm6NJkyYQQuDdd99VbXdxcUHt2rVx8+ZNjeMHDx4MR0dH1eM+ffqgUqVK2LNnDwAgNjYW165dw9tvv41Hjx4hOTkZycnJyMjIQPv27REZGQmFQqF2zg8//LBAte/ZswfNmjVTGybh4OCA999/H7du3cLFixcL9ibksn37dshkMsyaNUtj3+uGThSEchzy7t278fLly0Id+/jxYxw8eBD9+vVDenq66r189OgRwsLCcO3aNY3ZQkaMGAFzc3ONGk6cOGGSY8qJyjoGYCIqcz799FNkZ2e/dixwYVSpUkXtsbOzM2xsbODq6qqx/cmTJxrH+/r6qj2WyWSoWbOmaszrtWvXAABDhgyBm5ub2u27775DZmYmUlNT1c5RrVq1AtV++/Zt1K5dW2N73bp1VfsL68aNG/D09ET58uULfWxBhISEoHfv3pgzZw5cXV3Ro0cPrFu3DpmZma899vr16xBCYMaMGRrvpTKwP3jwQO0Ybe/lggULcP78eXh7e6NZs2aYPXu21l9uiKj04RhgIipzqlevjnfeeQfffPMNpk2bprE/rx5KuVye5zlf7R3MaxuAfMfj5kXZu7tw4UIEBgZqbePg4KD2WDl2uDQp6HuvXDwkOjoau3btwr59+zB8+HAsXrwY0dHRGu9Fbsr38uOPP0ZYWJjWNjVr1lR7rO297NevH1q3bo2dO3fir7/+wsKFCzF//nzs2LEDXbp0yfd1EpFxYwAmojLp008/xU8//YT58+dr7CtXrhwAICUlRW27Lj2hBaXs4VUSQuD69euqi7Zq1KgBAHByckKHDh30+txVq1bFlStXNLZfvnxZtb+watSogX379uHx48eF6gXO/d7nnm4tr/e+efPmaN68Ob744gv8/PPPGDhwIDZv3oz33nsvzzBdvXp1AIClpWWR38tKlSph1KhRGDVqFB48eIBGjRrhiy++YAAmKuU4BIKIyqQaNWrgnXfewdq1a5GYmKi2z8nJCa6uroiMjFTb/vXXXxdbPT/++CPS09NVj7dt24aEhARVkGrcuDFq1KiBRYsW4enTpxrHP3z4UOfn7tq1K06ePImoqCjVtoyMDHzzzTfw8fGBn59foc/Zu3dvCCFUC1Hkll8PuDLo537vMzIysGHDBrV2T5480TiPsmdcOQzCzs4OgOYvMu7u7mjbti3Wrl2rdRx4Qd5LuVyuMeTE3d0dnp6eBRqGQUTGjT3ARFRm/ec//8HGjRtx5coV+Pv7q+177733MG/ePLz33nto0qQJIiMjcfXq1WKrpXz58mjVqhWGDRuGpKQkLF26FDVr1sSIESMAAGZmZvjuu+/QpUsX+Pv7Y9iwYahcuTLu3buHQ4cOwcnJCbt27dLpuadNm4ZNmzahS5cuGDt2LMqXL48NGzYgLi4O27dvh5lZ4ftCQkNDMWjQICxfvhzXrl1D586doVAocOTIEYSGhmLMmDFaj+vUqROqVKmCd999F5MnT4a5uTl++OEHuLm5IT4+XtVuw4YN+Prrr/HWW2+hRo0aSE9Px7fffgsnJyd07doVgDRswc/PD1u2bEGtWrVQvnx51KtXD/Xq1cOqVavQqlUr1K9fHyNGjED16tWRlJSEqKgo3L17F2fPns339aWnp8PLywt9+vRBgwYN4ODggL///hunTp3Kc65nIio9GICJqMyqWbMm3nnnHY3eRQCYOXMmHj58iG3btuGXX35Bly5d8Oeff8Ld3b1Yavnkk0/w77//Yu7cuUhPT0f79u3x9ddfq3oxAWk536ioKPz3v//FypUr8fTpU3h4eCAoKAgffPCBzs9dsWJFHD9+HFOnTsWKFSvw4sULBAQEYNeuXejWrZvO5123bh0CAgLw/fffY/LkyXB2dkaTJk3QokWLPI+xtLTEzp07MWrUKMyYMQMeHh4YP348ypUrh2HDhqnahYSE4OTJk9i8eTOSkpLg7OyMZs2a4X//+5/aBWvfffcdPvroI0yYMAFZWVmYNWsW6tWrBz8/P5w+fRpz5szB+vXr8ejRI7i7u6Nhw4aYOXPma1+bnZ0dRo0ahb/++gs7duyAQqFAzZo18fXXX2PkyJE6v2dEZBxkQperNYiIiIiISimOASYiIiIik8IATEREREQmhQGYiIiIiEwKAzARERERmRQGYCIiIiIyKQzARPR/7daBAAAAAIAgf+tBLooAYEWAAQBYEWAAAFYEGACAFQEGAGBFgAEAWBFgAABWBBgAgJUAmg7gZZcpejUAAAAASUVORK5CYII=", "text/plain": [ "

" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "y = mean_rejection_data\n", "x = y.index.values\n", "\n", "plt.rcParams[\"figure.figsize\"] = (8, 5)\n", "plt.xlabel(\"Number of clusters\", fontsize=12)\n", "plt.ylabel(\"Rejection rate\", fontsize=12)\n", "plt.plot(x, y[\"uniform\"], label=\"Uniform Bootstrap\", color=\"blue\", marker=\"o\")\n", "plt.plot(x, y[\"cluster\"], label=\"Cluster Bootstrap\", color=\"red\", marker=\"o\")\n", "plt.legend()\n", "plt.suptitle(\"Comparison of Rejection Rates\", fontsize=15)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that when the number of clusters is low, it is particularly important to use the cluster robust bootstrap, since rejection with the regular bootstrap is excessive. For a large number of clusters, clustering naturally becomes less important. " ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.8" }, "vscode": { "interpreter": { "hash": "e8a16b1bdcc80285313db4674a5df2a5a80c75795379c5d9f174c7c712f05b3a" } } }, "nbformat": 4, "nbformat_minor": 4 } (robust_likelihood_inference)= # Robust Likelihood inference (to be written.) In case of an urgent request for this guide, feel free to open an issue \[here\](). # estimagic API ```{eval-rst} .. currentmodule:: estimagic ``` (estimation)= ## Estimation ```{eval-rst} .. dropdown:: estimate_ml .. autofunction:: estimate_ml ``` ```{eval-rst} .. dropdown:: estimate_msm .. autofunction:: estimate_msm ``` ```{eval-rst} .. dropdown:: get_moments_cov .. autofunction:: get_moments_cov ``` ```{eval-rst} .. dropdown:: lollipop_plot .. autofunction:: lollipop_plot ``` ```{eval-rst} .. dropdown:: estimation_table .. autofunction:: estimation_table ``` ```{eval-rst} .. dropdown:: render_html .. autofunction:: render_html ``` ```{eval-rst} .. dropdown:: render_latex .. autofunction:: render_latex ``` ```{eval-rst} .. dropdown:: LikelihoodResult .. autoclass:: LikelihoodResult :members: ``` ```{eval-rst} .. dropdown:: MomentsResult .. autoclass:: MomentsResult :members: ``` (bootstrap)= ## Bootstrap ```{eval-rst} .. dropdown:: bootstrap .. autofunction:: bootstrap ``` ```{eval-rst} .. dropdown:: BootstrapResult .. autoclass:: BootstrapResult :members: ``` # Installation ## Basic installation The preferred way to install optimagic is via `conda` or `mamba`. To do so, open a terminal and type: ``` conda install -c conda-forge optimagic ``` Alternatively, you can install optimagic via pip: ``` pip install optimagic ``` In both cases, you get optimagic and all of its mandatory dependencies. ## Installing optional dependencies Only `scipy` is a mandatory dependency of optimagic. Other algorithms become available if you install more packages. We make this optional because you will rarely need all of them in the same project. For an overview of all optimizers and the packages you need to install to enable them, see {ref}`list_of_algorithms`. To enable all algorithms at once, do the following: ``` conda -c conda-forge install nlopt ``` ``` pip install Py-BOBYQA ``` ``` pip install DFO-LS ``` *Note*: We recommend to install `DFO-LS` version 1.5.3 or higher. Versions of 1.5.0 or lower also work but the versions `1.5.1` and `1.5.2` contain bugs that can lead to errors being raised. ``` conda install -c conda-forge petsc4py ``` *Note*: `` `petsc4py` `` is not available on Windows. ``` conda install -c conda-forge cyipopt ``` *Note*: Make sure you have at least `cyipopt` 1.4. ``` conda install -c conda-forge pygmo ``` ``` pip install fides>=0.7.4 ``` *Note*: Make sure you have at least `fides` 0.7.4.