I've written several frameworks for optimization and the approach here of defining special objective functions works quite well. I like the look of this and would love to try it out soon.
This is a small thing, but I'm really happy to see that their L-BFGS line search algorithm supports a maximum step size. Many numerical optimization libraries do not offer this simple feature, which is critical if your goal is to find a minimum that is close to the initial starting location. This is important in atomistic geometry optimization.
This is looking very nice, can't wait to try it out. But I feel it's a missed opportunity to not provide some simple examples [1], as well as a benchmark to quickly estimate the amount of flops that can be expected. The main reason to use a library like this one over pyTorch or autograd/scipy is speed, after all.
[1] The test cases can be seen as examples, obviously, but they are not written in an easily accessible form IMHO.
EDIT: Looking more closely, it seems you need to provide an explicit gradient, no autodiff included, which makes it a complete nonstarter for me.
Looked at the documentation, had to do a double take "Is this mlpack??? a fork??" then checked, yup, it's a header only version of the mlpack's optimization library. Very cool.
Ceres is great for sparse, structured non-linear-least-squares problems (that is, problems where levenberg-marquardt is applicable). I believe it was initially designed for bundle-adjustment problems arising from structure-from-motion applications.
Ceres also does work for dense problems and general convex optimization, but it doesn't provide as much of an advantage over other packages (that is, you could just call into any existing l-bfgs implementation and wouldn't benefit much).
For some other problems, Ceres doesn't scale or can get in the way. For example, if you have a single value to maximize and a large number of variables, then ceres' auto-diff system won't work (ceres::Jet is based on "forward-mode" automatic differentiation, storing partial derivatives on the stack. If you have a high-dimensional gradient, it can exhaust stack space). On the other hand, Ensmallen doesn't seem to come with any auto-diff functionality.
As far as I can see, Ceres accepts functions which calculate a residual vector, whereas ensmallen accepts only function which accept a scalar error.
This is a subtle but important difference: The vector allows much faster convergence on a number of model fitting problems.
ensmallen also has support for batch optimization (like SGD), constrained problems, and SDPs for instance. The set of implemented optimizers is also quite large (40+ algorithms).
This is a small thing, but I'm really happy to see that their L-BFGS line search algorithm supports a maximum step size. Many numerical optimization libraries do not offer this simple feature, which is critical if your goal is to find a minimum that is close to the initial starting location. This is important in atomistic geometry optimization.