Ceres solver中几种常用算法介绍

http://homes.cs.washington.edu/~sagarwal/ceres-solver/stable/solving.html

Solving

Introduction

Effective use of Ceres requires some familiarity with the basic components of a nonlinear least squares solver, so before we describe how to configure and use the solver, we will take a brief look at how some
of the core optimization algorithms in Ceres work.
Let x∈Rn be
an n-dimensional
vector of variables, and F(x)=[f1(x),...,fm(x)]⊤ be
a m-dimensional
function of x.
We are interested in solving the following optimization problem [1] .
(1)

argminx12∥F(x)∥2 .

Here, the Jacobian J(x) of F(x) is
an m×n matrix,
where Jij(x)=∂jfi(x) and
the gradient vector g(x)=∇12∥F(x)∥2=J(x)⊤F(x).
Since the efficient global minimization of (1) for
general F(x) is
an intractable problem, we will have to settle for finding a local minimum.
The general strategy when solving non-linear optimization problems is to solve a sequence of approximations to the original problem [NocedalWright].
At each iteration, the approximation is solved to determine a correction Δx to
the vector x.
For non-linear least squares, an approximation can be constructed by using the linearization F(x+Δx)≈F(x)+J(x)Δx,
which leads to the following linear least squares problem:
(2)

minΔx12∥J(x)Δx+F(x)∥2

Unfortunately, naively solving a sequence of these problems and updating x←x+Δx leads
to an algorithm that may not converge. To get a convergent algorithm, we need to control the size of the step Δx.
Depending on how the size of the step Δx is
controlled, non-linear optimization algorithms can be divided into two major categories [NocedalWright].

Trust Region The trust region approach approximates the objective function using using a model function (often a quadratic) over a subset of the search space known as the trust region. If the model function succeeds in minimizing the true
objective function the trust region is expanded; conversely, otherwise it is contracted and the model optimization problem is solved again.
Line Search The line search approach first finds a descent direction along which the objective function will be reduced and then computes a step size that decides how far should move along that direction. The descent direction can be computed
by various methods, such as gradient descent, Newton’s method and Quasi-Newton method. The step size can be determined either exactly or inexactly.

Trust region methods are in some sense dual to line search methods: trust region methods first choose a step size (the size of the trust region) and then a step direction while line search methods first choose
a step direction and then a step size. Ceres implements multiple algorithms in both categories.

Trust Region Methods

The basic trust region algorithm looks something like this.

Given an initial point x and
a trust region radius μ.
argminΔx12∥J(x)Δx+F(x)∥2 s.t. ∥D(x)Δx∥2≤μ
ρ=∥F(x+Δx)∥2−∥F(x)∥2∥J(x)Δx+F(x)∥2−∥F(x)∥2
if ρ>ϵ then x=x+Δx.
if ρ>η1 then ρ=2ρ
else if ρ<η2 then ρ=0.5∗ρ
Goto 2.

Here, μ is
the trust region radius, D(x) is
some matrix used to define a metric on the domain of F(x) and ρ measures
the quality of the step Δx,
i.e., how well did the linear model predict the decrease in the value of the non-linear objective. The idea is to increase or decrease the radius of the trust region depending on how well the linearization predicts the behavior of the non-linear objective,
which in turn is reflected in the value of ρ.
The key computational step in a trust-region algorithm is the solution of the constrained optimization problem
(3)

argminΔx12∥J(x)Δx+F(x)∥2such
that∥D(x)Δx∥2≤μ

There are a number of different ways of solving this problem, each giving rise to a different concrete trust-region algorithm. Currently Ceres, implements two trust-region algorithms - Levenberg-Marquardt and
Dogleg. The user can choose between them by setting Solver::Options::trust_region_strategy_type.

Footnotes

[1]	At the level of the non-linear solver, the block structure is not relevant, therefore our discussion here is in terms of an optimization problem defined over a state vector of size n.

Levenberg-Marquardt

The Levenberg-Marquardt algorithm [Levenberg] [Marquardt] is
the most popular algorithm for solving non-linear least squares problems. It was also the first trust region algorithm to be developed [Levenberg] [Marquardt].
Ceres implements an exact step[Madsen] and
an inexact step variant of the Levenberg-Marquardt algorithm [WrightHolt] [NashSofer].
It can be shown, that the solution to (3) can
be obtained by solving an unconstrained optimization of the form

argminΔx12∥J(x)Δx+F(x)∥2+λ∥D(x)Δx∥2

Where, λ is
a Lagrange multiplier that is inverse related to μ.
In Ceres, we solve for
(4)

argminΔx12∥J(x)Δx+F(x)∥2+1μ∥D(x)Δx∥2

The matrix D(x) is
a non-negative diagonal matrix, typically the square root of the diagonal of the matrix J(x)⊤J(x).
Before going further, let us make some notational simplifications. We will assume that the matrix μ−−√D has
been concatenated at the bottom of the matrix J and
similarly a vector of zeros has been added to the bottom of the vector f and
the rest of our discussion will be in terms of J and f,
i.e, the linear least squares problem.
(5)

minΔx12∥J(x)Δx+f(x)∥2.

For all but the smallest problems the solution of (5) in
each iteration of the Levenberg-Marquardt algorithm is the dominant computational cost in Ceres. Ceres provides a number of different options for solving (5).
There are two major classes of methods - factorization and iterative.
The factorization methods are based on computing an exact solution of (4) using
a Cholesky or a QR factorization and lead to an exact step Levenberg-Marquardt algorithm. But it is not clear if an exact solution of (4) is
necessary at each step of the LM algorithm to solve (1).
In fact, we have already seen evidence that this may not be the case, as (4) is
itself a regularized version of (2).
Indeed, it is possible to construct non-linear optimization algorithms in which the linearized problem is solved approximately. These algorithms are known as inexact Newton or truncated Newton methods [NocedalWright].
An inexact Newton method requires two ingredients. First, a cheap method for approximately solving systems of linear equations. Typically an iterative linear solver like the Conjugate Gradients method is used
for this purpose [NocedalWright].
Second, a termination rule for the iterative solver. A typical termination rule is of the form
(6)

∥H(x)Δx+g(x)∥≤ηk∥g(x)∥.

Here, k indicates
the Levenberg-Marquardt iteration number and 0<ηk<1 is
known as the forcing sequence. [WrightHolt]prove
that a truncated Levenberg-Marquardt algorithm that uses an inexact Newton step based on (6) converges
for any sequence ηk≤η0<1 and
the rate of convergence depends on the choice of the forcing sequence ηk.
Ceres supports both exact and inexact step solution strategies. When the user chooses a factorization based linear solver, the exact step Levenberg-Marquardt algorithm is used. When the user chooses an iterative
linear solver, the inexact step Levenberg-Marquardt algorithm is used.

Dogleg

Another strategy for solving the trust region problem (3) was
introduced by M. J. D. Powell. The key idea there is to compute two vectors

ΔxGauss-NewtonΔxCauchy=argminΔx12∥J(x)Δx+f(x)∥2.=−∥g(x)∥2∥J(x)g(x)∥2g(x).

Note that the vector ΔxGauss-Newton is
the solution to (2) and ΔxCauchy is
the vector that minimizes the linear approximation if we restrict ourselves to moving along the direction of the gradient. Dogleg methods finds a vector Δx defined
by ΔxGauss-Newton and ΔxCauchy that
solves the trust region problem. Ceres supports two variants that can be chose by setting Solver::Options::dogleg_type.
TRADITIONAL_DOGLEG as
described by Powell, constructs two line segments using the Gauss-Newton and Cauchy vectors and finds the point farthest along this line shaped like a dogleg (hence the name) that is contained in the trust-region. For more details on the exact reasoning and
computations, please see Madsen et al [Madsen].
SUBSPACE_DOGLEG is
a more sophisticated method that considers the entire two dimensional subspace spanned by these two vectors and finds the point that minimizes the trust region problem in this subspace [ByrdSchnabel].
The key advantage of the Dogleg over Levenberg Marquardt is that if the step computation for a particular choice of μ does
not result in sufficient decrease in the value of the objective function, Levenberg-Marquardt solves the linear approximation from scratch with a smaller value of μ.
Dogleg on the other hand, only needs to compute the interpolation between the Gauss-Newton and the Cauchy vectors, as neither of them depend on the value of μ.
The Dogleg method can only be used with the exact factorization based linear solvers.

Inner Iterations

Some non-linear least squares problems have additional structure in the way the parameter blocks interact that it is beneficial to modify the way the trust region step is computed. e.g., consider the following
regression problem

y=a1eb1x+a2eb3x2+c1

Given a set of pairs {(xi,yi)},
the user wishes to estimate a1,a2,b1,b2,
and c1.
Notice that the expression on the left is linear in a1 and a2,
and given any value for b1,b2 and c1,
it is possible to use linear regression to estimate the optimal values of a1 and a2.
It’s possible to analytically eliminate the variables a1 and a2 from
the problem entirely. Problems like these are known as separable least squares problem and the most famous algorithm for solving them is the Variable Projection algorithm invented by Golub & Pereyra [GolubPereyra].
Similar structure can be found in the matrix factorization with missing data problem. There the corresponding algorithm is known as Wiberg’s algorithm [Wiberg].
Ruhe & Wedin present an analysis of various algorithms for solving separable non-linear least squares problems and refer toVariable Projection as Algorithm I in their paper [RuheWedin].
Implementing Variable Projection is tedious and expensive. Ruhe & Wedin present a simpler algorithm with comparable convergence properties, which they call Algorithm II. Algorithm II performs an additional optimization
step to estimate a1 and a2exactly
after computing a successful Newton step.
This idea can be generalized to cases where the residual is not linear in a1 and a2,
i.e.,

y=f1(a1,eb1x)+f2(a2,eb3x2+c1)

In this case, we solve for the trust region step for the full problem, and then use it as the starting point to further optimize just a_1and a_2.
For the linear case, this amounts to doing a single linear least squares solve. For non-linear problems, any method for solving the a_1 and a_2 optimization
problems will do. The only constraint on a_1 and a_2 (if
they are two different parameter block) is that they do not co-occur in a residual block.
This idea can be further generalized, by not just optimizing (a1,a2),
but decomposing the graph corresponding to the Hessian matrix’s sparsity structure into a collection of non-overlapping independent sets and optimizing each of them.
Setting Solver::Options::use_inner_iterations to true enables
the use of this non-linear generalization of Ruhe & Wedin’s Algorithm II. This version of Ceres has a higher iteration complexity, but also displays better convergence behavior per iteration.
Setting Solver::Options::num_threads to
the maximum number possible is highly recommended.

Non-monotonic Steps

Note that the basic trust-region algorithm described in Algorithm~ref{alg:trust-region} is a descent algorithm in that they only accepts a point if it strictly reduces the value of the objective function.
Relaxing this requirement allows the algorithm to be more efficient in the long term at the cost of some local increase in the value of the objective function.
This is because allowing for non-decreasing objective function values in a principled manner allows the algorithm to jump over boulders as the method is not restricted to move into narrow valleys while
preserving its convergence properties.
Setting Solver::Options::use_nonmonotonic_steps to true enables
the non-monotonic trust region algorithm as described by Conn, Gould & Toint in [Conn].
Even though the value of the objective function may be larger than the minimum value encountered over the course of the optimization, the final parameters returned to the user are the ones corresponding to the
minimum cost over all iterations.
The option to take non-monotonic steps is available for all trust region strategies.

Line Search Methods

The implementation of line search algorithms in Ceres Solver is fairly new and not very well tested, so for now this part of the solver should be considered beta quality. We welcome reports of your experiences
both good and bad on the mailinglist.
Line search algorithms

Given an initial point x
Δx=−H−1(x)g(x)
argminμ12∥F(x+μΔx)∥2
x=x+μΔx
Goto 2.

Here H(x) is
some approximation to the Hessian of the objective function, and g(x) is
the gradient at x.
Depending on the choice of H(x) we
get a variety of different search directions -Delta x.
Step 4, which is a one dimensional optimization or Line Search along Δx is
what gives this class of methods its name.
Different line search algorithms differ in their choice of the search direction Δx and
the method used for one dimensional optimization along Δx.
The choice of H(x) is
the primary source of computational complexity in these methods. Currently, Ceres Solver supports three choices of search directions, all aimed at large scale problems.

STEEPEST_DESCENT This
corresponds to choosing H(x) to
be the identity matrix. This is not a good search direction for anything but the simplest of the problems. It is only included here for completeness.
NONLINEAR_CONJUGATE_GRADIENT A
generalization of the Conjugate Gradient method to non-linear functions. The generalization can be performed in a number of different ways, resulting in a variety of search directions. Ceres Solver currently supports FLETCHER_REEVES, POLAK_RIBIRERE and HESTENES_STIEFEL directions.
BFGS A generalization
of the Secant method to multiple dimensions in which a full, dense approximation to the inverse Hessian is maintained and used to compute a quasi-Newton step [NocedalWright].
BFGS is currently the best known general quasi-Newton algorithm.
LBFGS A limited
memory approximation to the full BFGS method
in which the last M iterations are used to approximate the inverse Hessian used to compute a quasi-Newton step [Nocedal], [ByrdNocedal].

Currently Ceres Solver supports both a backtracking and interpolation based Armijo line search algorithm, and a sectioning / zoom interpolation (strong) Wolfe condition line search algorithm. However, note that
in order for the assumptions underlying theBFGS and LBFGS methods
to be guaranteed to be satisfied the Wolfe line search algorithm should be used.

LinearSolver

Recall that in both of the trust-region methods described above, the key computational cost is the solution of a linear least squares problem of the form
(7)

minΔx12∥J(x)Δx+f(x)∥2.

Let H(x)=J(x)⊤J(x) and g(x)=−J(x)⊤f(x).
For notational convenience let us also drop the dependence on x.
Then it is easy to see that solving (7) is
equivalent to solving the normal equations.
(8)

HΔx=g

Ceres provides a number of different options for solving (8).

DENSE_QR

For small problems (a couple of hundred parameters and a few thousand residuals) with relatively dense Jacobians, DENSE_QR is
the method of choice [Bjorck].
Let J=QR be
the QR-decomposition of J,
where Q is
an orthonormal matrix and R is
an upper triangular matrix [TrefethenBau].
Then it can be shown that the solution to (8) is
given by

Δx∗=−R−1Q⊤f

Ceres uses Eigen ‘s
dense QR factorization routines.

DENSE_NORMAL_CHOLESKY & SPARSE_NORMAL_CHOLESKY

Large non-linear least square problems are usually sparse. In such cases, using a dense QR factorization is inefficient. Let H=R⊤R be
the Cholesky factorization of the normal equations, where R is
an upper triangular matrix, then the solution to(8) is
given by

Δx∗=R−1R−⊤g.

The observant reader will note that the R in
the Cholesky factorization of H is
the same upper triangular matrix R in
the QR factorization of J.
Since Q is
an orthonormal matrix, J=QR implies
that J⊤J=R⊤Q⊤QR=R⊤R.
There are two variants of Cholesky factorization – sparse and dense.
DENSE_NORMAL_CHOLESKY as
the name implies performs a dense Cholesky factorization of the normal equations. Ceres usesEigen ‘s
dense LDLT factorization routines.
SPARSE_NORMAL_CHOLESKY,
as the name implies performs a sparse Cholesky factorization of the normal equations. This leads to substantial savings in time and memory for large sparse problems. Ceres uses the sparse Cholesky factorization routines in Professor Tim Davis’ SuiteSparse or CXSparse packages [Chen].

DENSE_SCHUR & SPARSE_SCHUR

While it is possible to use SPARSE_NORMAL_CHOLESKY to
solve bundle adjustment problems, bundle adjustment problem have a special structure, and a more efficient scheme for solving (8) can
be constructed.
Suppose that the SfM problem consists of p cameras
and q points
and the variable vector x has
the block structure x=[y1,...,yp,z1,...,zq].
Where, y and z correspond
to camera and point parameters, respectively. Further, let the camera blocks be of size c and
the point blocks be of size s (for
most problems c = 6–9 and s=3).
Ceres does not impose any constancy requirement on these block sizes, but choosing them to be constant simplifies the exposition.
A key characteristic of the bundle adjustment problem is that there is no term fi that
includes two or more point blocks. This in turn implies that the matrix H is
of the form
(9)

H=[BE⊤EC] ,

where, B∈Rpc×pc is
a block sparse matrix with p blocks
of size c×c and C∈Rqs×qs is
a block diagonal matrix with qblocks
of size s×s. E∈Rpc×qs is
a general block sparse matrix, with a block of size c×s for
each observation. Let us now block partition Δx=[Δy,Δz] and g=[v,w] to
restate (8) as
the block structured linear system
(10)

[BE⊤EC][ΔyΔz]=[vw] ,

and apply Gaussian elimination to it. As we noted above, C is
a block diagonal matrix, with small diagonal blocks of size s×s.
Thus, calculating the inverse of C by
inverting each of these blocks is cheap. This allows us to eliminate Δz by
observing that Δz=C−1(w−E⊤Δy),
giving us
(11)

[B−EC−1E⊤]Δy=v−EC−1w .

The matrix

S=B−EC−1E⊤

is the Schur complement of C in H.
It is also known as the reduced camera matrix, because the only variables participating in(11) are
the ones corresponding to the cameras. S∈Rpc×pc is
a block structured symmetric positive definite matrix, with blocks of size c×c.
The block Sij corresponding
to the pair of images i and j is
non-zero if and only if the two images observe at least one common point.
Now, eq-linear2 can be solved by first forming S,
solving for Δy,
and then back-substituting Δy to
obtain the value of Δz.
Thus, the solution of what was an n×n, n=pc+qs linear
system is reduced to the inversion of the block diagonal matrix C,
a few matrix-matrix and matrix-vector multiplies, and the solution of block sparse pc×pc linear
system (11).
For almost all problems, the number of cameras is much smaller than the number of points, p≪q,
thus solving (11) is
significantly cheaper than solving (10).
This is the Schur complement trick [Brown].
This still leaves open the question of solving (11).
The method of choice for solving symmetric positive definite systems exactly is via the Cholesky factorization [TrefethenBau] and
depending upon the structure of the matrix, there are, in general, two options. The first is direct factorization, where we store and factor S as
a dense matrix [TrefethenBau].
This method has O(p2) space
complexity and O(p3) time
complexity and is only practical for problems with up to a few hundred cameras. Ceres implements this strategy as the DENSE_SCHUR solver.
But, S is
typically a fairly sparse matrix, as most images only see a small fraction of the scene. This leads us to the second option: Sparse Direct Methods. These methods store S as
a sparse matrix, use row and column re-ordering algorithms to maximize the sparsity of the Cholesky decomposition, and focus their compute effort on the non-zero part of the factorization[Chen].
Sparse direct methods, depending on the exact sparsity structure of the Schur complement, allow bundle adjustment algorithms to significantly scale up over those based on dense factorization. Ceres implements this strategy as theSPARSE_SCHUR solver.

CGNR

For general sparse problems, if the problem is too large for CHOLMOD or
a sparse linear algebra library is not linked into Ceres, another option is the CGNR solver.
This solver uses the Conjugate Gradients solver on the normal equations, but without forming the normal equations explicitly. It exploits the relation

Hx=J⊤Jx=J⊤(Jx)

When the user chooses ITERATIVE_SCHUR as
the linear solver, Ceres automatically switches from the exact step algorithm to an inexact step algorithm.

ITERATIVE_SCHUR

Another option for bundle adjustment problems is to apply PCG to the reduced camera matrix S instead
of H.
One reason to do this is that S is
a much smaller matrix than H,
but more importantly, it can be shown that κ(S)≤κ(H).
Cseres implements PCG on S as
the ITERATIVE_SCHUR solver.
When the user chooses ITERATIVE_SCHUR as
the linear solver, Ceres automatically switches from the exact step algorithm to an inexact step algorithm.
The cost of forming and storing the Schur complement S can
be prohibitive for large problems. Indeed, for an inexact Newton solver that computes S and
runs PCG on it, almost all of its time is spent in constructing S;
the time spent inside the PCG algorithm is negligible in comparison. Because PCG only needs access to S via
its product with a vector, one way to evaluate Sx is
to observe that

x1=E⊤x

x2=C−1x1

x3=Ex2

x4=Bx

(12)

Sx=x4−x3

Thus, we can run PCG on S with
the same computational effort per iteration as PCG on H,
while reaping the benefits of a more powerful preconditioner. In fact, we do not even need to compute H, (12) can
be implemented using just the columns of J.
Equation (12) is
closely related to Domain Decomposition methods for solving large linear systems that arise in structural engineering and partial differential equations. In the language of Domain Decomposition, each point in a bundle adjustment problem is a domain,
and the cameras form the interface between these domains. The iterative solution of the Schur complement then falls within the sub-category of techniques known as Iterative Sub-structuring [Saad] [Mathew].

Preconditioner

The convergence rate of Conjugate Gradients for solving (8) depends
on the distribution of eigenvalues of H [Saad].
A useful upper bound is κ(H)−−−−√,
where, κ(H) is
the condition number of the matrix H.
For most bundle adjustment problems, κ(H) is
high and a direct application of Conjugate Gradients to (8) results
in extremely poor performance.
The solution to this problem is to replace (8) with
a preconditioned system. Given a linear system, Ax=b and
a preconditioner M the
preconditioned system is given by M−1Ax=M−1b.
The resulting algorithm is known as Preconditioned Conjugate Gradients algorithm (PCG) and its worst case complexity now depends on the condition number of the preconditioned matrix κ(M−1A).
The computational cost of using a preconditioner M is
the cost of computing M and
evaluating the product M−1y for
arbitrary vectors y.
Thus, there are two competing factors to consider: How much of H‘s
structure is captured by M so
that the condition number κ(HM−1) is
low, and the computational cost of constructing and using M.
The ideal preconditioner would be one for which κ(M−1A)=1. M=A achieves
this, but it is not a practical choice, as applying this preconditioner would require solving a linear system equivalent to the unpreconditioned problem. It is usually the case that the more information M has
about H,
the more expensive it is use. For example, Incomplete Cholesky factorization based preconditioners have much better convergence behavior than the Jacobi preconditioner, but are also much more expensive.
The simplest of all preconditioners is the diagonal or Jacobi preconditioner, i.e., M=diag(A),
which for block structured matrices like H can
be generalized to the block Jacobi preconditioner.
For ITERATIVE_SCHUR there
are two obvious choices for block diagonal preconditioners for S.
The block diagonal of the matrix B [Mandel] and
the block diagonal S,
i.e, the block Jacobi preconditioner for S.
Ceres’s implements both of these preconditioners and refers to them as JACOBI and SCHUR_JACOBI respectively.
For bundle adjustment problems arising in reconstruction from community photo collections, more effective preconditioners can be constructed by analyzing and exploiting the camera-point visibility structure of
the scene [KushalAgarwal]. Ceres implements the two visibility based preconditioners described by Kushal & Agarwal as CLUSTER_JACOBI and CLUSTER_TRIDIAGONAL.
These are fairly new preconditioners and Ceres’ implementation of them is in its early stages and is not as mature as the other preconditioners described above.

Ordering

The order in which variables are eliminated in a linear solver can have a significant of impact on the efficiency and accuracy of the method. For example when doing sparse Cholesky factorization, there are matrices
for which a good ordering will give a Cholesky factor with O(n) storage,
where as a bad ordering will result in an completely dense factor.
Ceres allows the user to provide varying amounts of hints to the solver about the variable elimination ordering to use. This can range from no hints, where the solver is free to decide the best ordering based
on the user’s choices like the linear solver being used, to an exact order in which the variables should be eliminated, and a variety of possibilities in between.
Instances of the ParameterBlockOrdering class
are used to communicate this information to Ceres.
Formally an ordering is an ordered partitioning of the parameter blocks. Each parameter block belongs to exactly one group, and each group has a unique integer associated with it, that determines its order in
the set of groups. We call these groupsElimination Groups
Given such an ordering, Ceres ensures that the parameter blocks in the lowest numbered elimination group are eliminated first, and then the parameter blocks in the next lowest numbered elimination group and so
on. Within each elimination group, Ceres is free to order the parameter blocks as it chooses. e.g. Consider the linear system

x+y2x+3y=3=7

There are two ways in which it can be solved. First eliminating x from
the two equations, solving for y and then back substituting for x,
or first eliminating y,
solving for x and
back substituting for y.
The user can construct three orderings here.

{0:x},{1:y} :
Eliminate x first.
{0:y},{1:x} :
Eliminate y first.
{0:x,y} :
Solver gets to decide the elimination order.

Thus, to have Ceres determine the ordering automatically using heuristics, put all the variables in the same elimination group. The identity of the group does not matter. This is the same as not specifying an
ordering at all. To control the ordering for every variable, create an elimination group per variable, ordering them in the desired order.
If the user is using one of the Schur solvers (DENSE_SCHUR, SPARSE_SCHUR, ITERATIVE_SCHUR)
and chooses to specify an ordering, it must have one important property. The lowest numbered elimination group must form an independent set in the graph corresponding to the Hessian, or in other words, no two parameter blocks in in the first elimination group
should co-occur in the same residual block. For the best performance, this elimination group should be as large as possible. For standard bundle adjustment problems, this corresponds to the first elimination group containing all the 3d points, and the second
containing the all the cameras parameter blocks.
If the user leaves the choice to Ceres, then the solver uses an approximate maximum independent set algorithm to identify the first elimination group [LiSaad].

Solver::Options

class Solver::Options
Solver::Options controls
the overall behavior of the solver. We list the various settings and their default values below.

MinimizerType Solver::Options::minimizer_type
Default: TRUST_REGION
Choose between LINE_SEARCH and TRUST_REGION algorithms.
See Trust
Region Methods and Line
Search Methods for more details.

LineSearchDirectionType Solver::Options::line_search_direction_type
Default: LBFGS
Choices are STEEPEST_DESCENT, NONLINEAR_CONJUGATE_GRADIENT, BFGS and LBFGS.

LineSearchType Solver::Options::line_search_type
Default: WOLFE
Choices are ARMIJO and WOLFE (strong
Wolfe conditions). Note that in order for the assumptions underlying the BFGS andLBFGS line
search direction algorithms to be guaranteed to be satisifed, the WOLFE line
search should be used.

NonlinearConjugateGradientType Solver::Options::nonlinear_conjugate_gradient_type
Default: FLETCHER_REEVES
Choices are FLETCHER_REEVES, POLAK_RIBIRERE and HESTENES_STIEFEL.

int Solver::Options::max_lbfs_rank
Default: 20
The L-BFGS hessian approximation is a low rank approximation to the inverse of the Hessian matrix. The rank of the approximation determines (linearly) the space and time complexity of using the approximation. Higher
the rank, the better is the quality of the approximation. The increase in quality is however is bounded for a number of reasons.

The method only uses secant information and not actual derivatives.
The Hessian approximation is constrained to be positive definite.

So increasing this rank to a large number will cost time and space complexity without the corresponding increase in solution quality. There are no hard and fast rules for choosing the maximum rank. The best choice
usually requires some problem specific experimentation.

bool Solver::Options::use_approximate_eigenvalue_bfgs_scaling

Default: false
As part of the BFGS update
step / LBFGS right-multiply
step, the initial inverse Hessian approximation is taken to be the Identity. However, [Oren] showed
that using instead I∗γ,
where γ is
a scalar chosen to approximate an eigenvalue of the true inverse Hessian can result in improved convergence in a wide variety of cases. Settinguse_approximate_eigenvalue_bfgs_scaling to
true enables this scaling in BFGS (before
first iteration) andLBFGS (at
each iteration).
Precisely, approximate eigenvalue scaling equates to

γ=y′ksky′kyk

With:

yk=∇fk+1−∇fk

sk=xk+1−xk

Where f() is
the line search objective and x the
vector of parameter values [NocedalWright].
It is important to note that approximate eigenvalue scaling does not always improve convergence, and that it can in factsignificantly degrade performance for certain classes of
problem, which is why it is disabled by default. In particular it can degrade performance when the sensitivity of the problem to different parameters varies significantly, as in this case a single scalar factor fails to capture this variation and detrimentally
downscales parts of the Jacobian approximation which correspond to low-sensitivity parameters. It can also reduce the robustness of the solution to errors in the Jacobians.

LineSearchIterpolationType Solver::Options::line_search_interpolation_type
Default: CUBIC
Degree of the polynomial used to approximate the objective function. Valid values are BISECTION, QUADRATIC and CUBIC.

double Solver::Options::min_line_search_step_size
The line search terminates if:

∥Δxk∥∞<min_line_search_step_size

where ∥⋅∥∞ refers
to the max norm, and Δxk is
the step change in the parameter values at the k-th
iteration.

double Solver::Options::line_search_sufficient_function_decrease
Default: 1e-4
Solving the line search problem exactly is computationally prohibitive. Fortunately, line search based optimization algorithms can still guarantee convergence if instead of an exact solution, the line search algorithm
returns a solution which decreases the value of the objective function sufficiently. More precisely, we are looking for a step size s.t.

f(step_size)≤f(0)+sufficient_decrease∗[f′(0)∗step_size]

This condition is known as the Armijo condition.

double Solver::Options::max_line_search_step_contraction
Default: 1e-3
In each iteration of the line search,

new_step_size>=max_line_search_step_contraction∗step_size

Note that by definition, for contraction:

0<max_step_contraction<min_step_contraction<1

double Solver::Options::min_line_search_step_contraction
Default: 0.6
In each iteration of the line search,

new_step_size<=min_line_search_step_contraction∗step_size

Note that by definition, for contraction:

0<max_step_contraction<min_step_contraction<1

int Solver::Options::max_num_line_search_step_size_iterations
Default: 20
Maximum number of trial step size iterations during each line search, if a step size satisfying the search conditions cannot be found within this number of trials, the line search will stop.
As this is an ‘artificial’ constraint (one imposed by the user, not the underlying math), if WOLFE line
search is being used, andpoints satisfying the Armijo sufficient (function) decrease condition have been found during the current search (in <=max_num_line_search_step_size_iterations).
Then, the step size with the lowest function value which satisfies the Armijo condition will be returned as the new valid step, even though it does not satisfy the strong Wolfe conditions. This behaviour protects against early termination of the optimizer
at a sub-optimal point.

int Solver::Options::max_num_line_search_direction_restarts
Default: 5
Maximum number of restarts of the line search direction algorithm before terminating the optimization. Restarts of the line search direction algorithm occur when the current algorithm fails to produce a new descent
direction. This typically indicates a numerical failure, or a breakdown in the validity of the approximations used.

double Solver::Options::line_search_sufficient_curvature_decrease
Default: 0.9
The strong Wolfe conditions consist of the Armijo sufficient decrease condition, and an additional requirement that the step size be chosen s.t. the magnitude (‘strong’ Wolfe conditions) of the gradient
along the search direction decreases sufficiently. Precisely, this second condition is that we seek a step size s.t.

∥f′(step_size)∥<=sufficient_curvature_decrease∗∥f′(0)∥

Where f() is
the line search objective and f′() is
the derivative of f with
respect to the step size: dfd step
size.

double Solver::Options::max_line_search_step_expansion
Default: 10.0
During the bracketing phase of a Wolfe line search, the step size is increased until either a point satisfying the Wolfe conditions is found, or an upper bound for a bracket containinqg a point satisfying the conditions
is found. Precisely, at each iteration of the expansion:

new_step_size<=max_step_expansion∗step_size

By definition for expansion

max_step_expansion>1.0

TrustRegionStrategyType Solver::Options::trust_region_strategy_type
Default: LEVENBERG_MARQUARDT
The trust region step computation algorithm used by Ceres. Currently LEVENBERG_MARQUARDT and DOGLEG are
the two valid choices. See Levenberg-Marquardt and Dogleg for
more details.

DoglegType Solver::Options::dogleg_type
Default: TRADITIONAL_DOGLEG
Ceres supports two different dogleg strategies. TRADITIONAL_DOGLEG method
by Powell and the SUBSPACE_DOGLEG method
described by [ByrdSchnabel] .
See Dogleg for
more details.

bool Solver::Options::use_nonmonotonic_steps
Default: false
Relax the requirement that the trust-region algorithm take strictly decreasing steps. See Non-monotonic
Steps for more details.

int Solver::Options::max_consecutive_nonmonotonic_steps
Default: 5
The window size used by the step selection algorithm to accept non-monotonic steps.

int Solver::Options::max_num_iterations
Default: 50
Maximum number of iterations for which the solver should run.

double Solver::Options::max_solver_time_in_seconds
Default: 1e6 Maximum
amount of time for which the solver should run.

int Solver::Options::num_threads
Default: 1
Number of threads used by Ceres to evaluate the Jacobian.

double Solver::Options::initial_trust_region_radius
Default: 1e4
The size of the initial trust region. When the LEVENBERG_MARQUARDT strategy
is used, the reciprocal of this number is the initial regularization parameter.

double Solver::Options::max_trust_region_radius
Default: 1e16
The trust region radius is not allowed to grow beyond this value.

double Solver::Options::min_trust_region_radius
Default: 1e-32
The solver terminates, when the trust region becomes smaller than this value.

double Solver::Options::min_relative_decrease
Default: 1e-3
Lower threshold for relative decrease before a trust-region step is accepted.

double Solver::Options::min_lm_diagonal
Default: 1e6
The LEVENBERG_MARQUARDT strategy,
uses a diagonal matrix to regularize the the trust region step. This is the lower bound on the values of this diagonal matrix.

double Solver::Options::max_lm_diagonal
Default: 1e32
The LEVENBERG_MARQUARDT strategy,
uses a diagonal matrix to regularize the the trust region step. This is the upper bound on the values of this diagonal matrix.

int Solver::Options::max_num_consecutive_invalid_steps
Default: 5
The step returned by a trust region strategy can sometimes be numerically invalid, usually because of conditioning issues. Instead of crashing or stopping the optimization, the optimizer can go ahead and try solving
with a smaller trust region/better conditioned problem. This parameter sets the number of consecutive retries before the minimizer gives up.

double Solver::Options::function_tolerance
Default: 1e-6
Solver terminates if

|Δcost|cost<function_tolerance

where, Δcost is
the change in objective function value (up or down) in the current iteration of Levenberg-Marquardt.

double Solver::Options::gradient_tolerance
Default: 1e-10
Solver terminates if