Reference

DEFAULT_HSL_DENSE_MAX_NVAR

Default dense_max_nvar of MA57R2NSubsolver and MA97R2NSubsolver: the largest nvar for which a fully dense Hessian is still handed to HSL.

DEFAULT_QRMUMPS_DENSE_MAX_ENTRIES

Default dense_max_entries of QRMumpsR2NLSSubsolver: the largest dense Jacobian, in number of entries, still handed to qrmumps. Set to `DEFAULTHSLDENSEMAXNVAR^2, i.e. a dense12000 × 12_000` Jacobian, so both direct subsolvers accept dense problems of comparable size.

Unlike the Hessian, the Jacobian is rectangular, so the threshold is on m * n rather than on nvar alone: a dense m = 1_000_000, n = 200 Jacobian is 2 × 10⁸ entries and is caught even though n is small.

AbstractR2NLSSubsolver{T}

Abstract type for subsolvers used in the R2NLS (Nonlinear Least Squares) algorithm.

AbstractR2NSubsolver{T}

Abstract type for subsolvers used in the R2N (Newton-type) algorithm.

AbstractSubsolver{T}

Base abstract type for all subproblem solvers.

Interface

Subtypes of AbstractSubsolver must be callable (act as functors) to solve the subproblem: (subsolver::AbstractSubsolver)(s, rhs, σ, atol, rtol; verbose=0)

FOMOParameterSet{T} <: AbstractParameterSet

This structure designed for fomo regroups the following parameters:

• η1, η2: step acceptance parameters.
• γ1, γ2: regularization update parameters.
• γ3 : momentum factor βmax update parameter in case of unsuccessful iteration.
• αmax: maximum step parameter for fomo algorithm.
• α0: initial value of the step parameter (capped by αmax); if 0, it is chosen from the initial gradient norm.
• β ∈ [0,1): target decay rate for the momentum.
• θ1: momentum contribution parameter for convergence condition (1).
• θ2: momentum contribution parameter for convergence condition (2).
• M : requires objective decrease over the M last iterates (nonmonotone context). M=1 implies monotone behaviour.
• step_backend: step computation mode. Options are r2_step() for quadratic regulation step and tr_step() for first-order trust-region.

An additional constructor is

FOMOParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• η1::T = eps(T)^(1 // 4)
• η2::T = T(95/100)
• γ1::T = T(1/2)
• γ2::T = T(2)
• γ3::T = T(1/2)
• αmax::T = 1/eps(T)
• α0::T = T(0)
• β = T(9/10) ∈ [0,1)
• θ1 = T(1/10)
• θ2 = eps(T)^(1/3)
• M = 1
• `stepbackend = r2step()

fo(nlp; kwargs...)
R2(nlp; kwargs...)
TR(nlp; kwargs...)

A First-Order (FO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FoSolver(nlp)
solve!(solver, nlp; kwargs...)

R2 and TR runs fo with the dedicated step_backend keyword argument.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• η1 = eps(T)^(1 // 4): algorithm parameter, see FOMOParameterSet.
• η2 = T(95/100): algorithm parameter, see FOMOParameterSet.
• γ1 = T(1/2): algorithm parameter, see FOMOParameterSet.
• γ2 = T(2): algorithm parameter, see FOMOParameterSet.
• αmax = 1/eps(T): algorithm parameter, see FOMOParameterSet.
• α0 = T(0): algorithm parameter, see FOMOParameterSet.
• max_eval::Int = -1: maximum number of evaluation of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• M = 1 : algorithm parameter, see FOMOParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): algorithm parameter, see FOMOParameterSet.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fo(nlp) # run with step_backend = r2_step(), equivalent to R2(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

fomo(nlp; kwargs...)

A First-Order with MOmentum (FOMO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

Algorithm description

The step is computed along d = - (1-βmax) .* ∇f(xk) - βmax .* mk with mk the memory of past gradients (initialized at 0), and updated at each successful iteration as mk .= ∇f(xk) .* (1 - βmax) .+ mk .* βmax and βmax ∈ [0,β] chosen as to ensure d is gradient-related, i.e., the following 2 conditions are satisfied: (1-βmax) .* ∇f(xk) + βmax .* ∇f(xk)ᵀmk ≥ θ1 * ‖∇f(xk)‖² (1) ‖∇f(xk)‖ ≥ θ2 * ‖(1-βmax) . ∇f(xk) + βmax . mk‖ (2) In the nonmonotone case, (1) rewrites (1-βmax) .* ∇f(xk) + βmax .* ∇f(xk)ᵀmk + (fm - fk)/μk ≥ θ1 * ‖∇f(xk)‖², with fm the largest objective value over the last M successful iterations, and fk = f(xk).

Advanced usage

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FomoSolver(nlp)
solve!(solver, nlp; kwargs...)

No momentum: if the user does not whish to use momentum (β = 0), it is recommended to use the memory-optimized fo method.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• η1 = eps(T)^(1 // 4), η2 = T(95/100): step acceptance parameters.
• γ1 = T(1/2), γ2 = T(2): regularization update parameters.
• γ3 = T(1/2) : momentum factor βmax update parameter in case of unsuccessful iteration.
• αmax = 1/eps(T): maximum step parameter for fomo algorithm.
• α0 = T(0): initial value of the step parameter (capped by αmax); if 0, it is chosen from the initial gradient norm.
• max_eval::Int = -1: maximum number of objective evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• β = T(9/10) ∈ [0,1): target decay rate for the momentum.
• θ1 = T(1/10): momentum contribution parameter for convergence condition (1).
• θ2 = eps(T)^(1/3): momentum contribution parameter for convergence condition (2).
• M = 1 : requires objective decrease over the M last iterates (nonmonotone context). M=1 implies monotone behaviour.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): step computation mode. Options are r2_step() for quadratic regulation step and tr_step() for first-order trust-region.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

fomo

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fomo(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FomoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

LBFGSParameterSet{T} <: AbstractParameterSet

This structure designed for lbfgs regroups the following parameters:

• mem: memory parameter of the lbfgs algorithm
• τ₁: slope factor in the Wolfe condition when performing the line search
• bk_max: maximum number of backtracks when performing the line search.

An additional constructor is

LBFGSParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• mem::Int = 5
• τ₁::T = T(0.9999)
• bk_max:: Int = 25

lbfgs(nlp; kwargs...)

An implementation of a limited memory BFGS line-search method for unconstrained minimization.

For advanced usage, first define a LBFGSSolver to preallocate the memory used in the algorithm, and then call solve!.

solver = LBFGSSolver(nlp; mem::Int = 5)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• mem::Int = 5: algorithm parameter, see LBFGSParameterSet.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• τ₁::T = T(0.9999): algorithm parameter, see LBFGSParameterSet.
• bk_max:: Int = 25: algorithm parameter, see LBFGSParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• verbose_subsolver::Int = 0: if > 0, display iteration information every verbose_subsolver iteration of the subsolver.

Output

The returned value is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3));
stats = lbfgs(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3));
solver = LBFGSSolver(nlp; mem = 5);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

QRMumpsR2NLSSubsolver(nls; fill_ratio = 0.5,
                      dense_max_entries = 12_000^2,
                      max_nvar = typemax(Int), max_nnzj = typemax(Int))

Direct sparse subsolver for R2NLS backed by qr_mumps.

Before touching qr_mumps, a cheap input-size and density guard is applied. If any of

• nls.meta.nvar > max_nvar,
• nls.nls_meta.nnzj > max_nnzj (nnzj counts stored Jacobian nonzeros), or
• nnzj > fill_ratio * m * n when m * n > dense_max_entries

is true, the constructor returns an "unsupported" placeholder: no qrmumps object is allocated, no symbolic analysis is run, and [`isunsupported](@ref) returnstrue.R2NLSthen short-circuits to status:exception`. Use this to avoid the eager symbolic analysis on problems that are too big to factorize.

Density guard and problem size

A dense Jacobian is cheap enough to factorize at moderate size, so the fill_ratio density test only applies once m * n > dense_max_entries: any problem with m * n ≤ dense_max_entries is handed to qrmumps no matter how dense it is. The threshold is on entries rather than on nvar because the Jacobian is m × n: a dense `m = 1000_000,n = 200Jacobian is 2 × 10⁸ entries (~1.6 GB inFloat64) and must be caught even thoughn` is small.

With the default dense_max_entries = 12_000^2, a dense m = n = 5_000 problem still runs through qrmumps. Set it to the largest dense Jacobian you accept factorizing; `densemaxentries = 0restores the "reject every dense Jacobian" behaviour, whilemaxnvar/max_nnzj` stay hard caps that apply regardless of density.

Negative max_nvar, max_nnzj or dense_max_entries raise ArgumentError.

R2NLSParameterSet([T=Float64]; η1, η2, θ1, θ2, γ1, γ2, γ3, δ1, σmin, nonmonosize)

Parameter set for the R2NLS solver. Controls algorithmic tolerances and step acceptance.

Keyword Arguments

• η1 = eps(T)^(1/4): Accept step if actual/predicted reduction ≥ η1 (0 < η1 ≤ η2 < 1).
• η2 = T(0.95): Step is very successful if reduction ≥ η2 (0 < η1 ≤ η2 < 1).
• θ1 = T(0.9): Controls Cauchy step size (0 < θ1 < 1).
• θ2 = eps(T)^(-1): Maximum allowed ratio between the step and the Cauchy step (θ2 > 1).
• γ1 = T(1.5): Regularization increase factor on successful (but not very successful) step (1 < γ1 ≤ γ2).
• γ2 = T(2.5): Regularization increase factor on rejected step (γ1 ≤ γ2).
• γ3 = T(0.5): Regularization decrease factor on very successful step (0 < γ3 ≤ 1).
• δ1 = T(0.5): Cauchy point scaling (0 < δ1 < 1). θ1 scales the step size when using the exact Cauchy point, while δ1 scales the step size inexact Cauchy point.
• σmin = eps(T): Smallest allowed regularization.
• non_mono_size = 1: Window size for non-monotone acceptance.
• compute_cauchy_point = false: Whether to compute the Cauchy point.
• inexact_cauchy_point = true: Whether to use an inexact Cauchy point.

R2NParameterSet([T=Float64]; θ1, θ2, η1, η2, γ1, γ2, γ3, σmin, non_mono_size)

Parameter set for the R2N solver. Controls algorithmic tolerances and step acceptance.

Keyword Arguments

• θ1 = T(0.9): Cauchy step parameter (0 < θ1 < 1).
• θ2 = eps(T)^(-1): Maximum allowed ratio between the step and the Cauchy step (θ2 > 1).
• η1 = eps(T)^(1/4): Accept step if actual/predicted reduction ≥ η1 (0 < η1 ≤ η2 < 1).
• η2 = T(0.95): Step is very successful if reduction ≥ η2 (0 < η1 ≤ η2 < 1).
• γ1 = T(1.5): Regularization increase factor on successful (but not very successful) step (1 < γ1 ≤ γ2).
• γ2 = T(2.5): Regularization increase factor on rejected step (γ1 ≤ γ2).
• γ3 = T(0.5): Regularization decrease factor on very successful step (0 < γ3 ≤ 1).
• δ1 = T(0.5): Cauchy point calculation parameter.
• σmin = eps(T): Minimum regularization parameter.
• non_mono_size = 1: Window size for non-monotone acceptance.
• bk_max = 10: Maximum number of backtracking (Armijo) steps in the NPC line search.
• bG_max = 10: Maximum number of Goldstein (extrapolation) steps in the NPC line search.

R2N(nlp; kwargs...)

A second-order quadratic regularization method for unconstrained optimization (with shifted L-BFGS or shifted Hessian operator).

min f(x)

For advanced usage, first define a R2NSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = R2NSolver(nlp)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• η1::T = eps(T)^(1/4): step acceptance parameter, see R2NParameterSet.
• η2::T = T(0.95): step acceptance parameter, see R2NParameterSet.
• θ1::T = T(0.9): Cauchy step parameter, see R2NParameterSet.
• θ2::T = eps(T)^(-1): Cauchy step parameter, see R2NParameterSet.
• γ1::T = T(1.5): regularization update parameter, see R2NParameterSet.
• γ2::T = T(2.5): regularization update parameter, see R2NParameterSet.
• γ3::T = T(0.5): regularization update parameter, see R2NParameterSet.
• δ1::T = T(0.5): Cauchy point calculation parameter, see R2NParameterSet.
• σmin::T = eps(T): minimum step parameter, see R2NParameterSet.
• non_mono_size::Int = 1: the size of the non-monotone behaviour. If > 1, the algorithm will use a non-monotone strategy to accept steps.
• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• max_eval::Int = -1: maximum number of evaluations of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver = CGR2NSubsolver: the subproblem solver type or instance.
  • Note: HSL guards fill_ratio, dense_max_nvar, max_nvar, and max_nnzh must be passed to a pre-built subsolver, e.g. subsolver = MA57R2NSubsolver(nlp; max_nvar = 100_000, max_nnzh = 5_000_000). max_nvar is compared against nlp.meta.nvar and max_nnzh against nlp.meta.nnzh (the number of stored lower-triangular Hessian nonzeros). Both default to typemax(Int) (no additional limit). The fill_ratio density guard only applies once nvar > dense_max_nvar (default 12_000), so dense problems up to that size are still factorized by HSL. If any guard trips, HSL allocation and symbolic analysis are skipped and R2N returns status :exception. These are cheap input-size checks, not bounds on factorization memory or runtime.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver if KrylovWorkspace type is selected.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• scp_flag::Bool = false: if true, we compare the norm of the calculate step with θ2 * norm(scp), each iteration, selecting the smaller step.
• always_accept_npc_ag::Bool = false: if true, we skip the computation of the reduction ratio ρ for Goldstein steps taken along directions of negative curvature, unconditionally accepting them as successful steps to aggressively escape saddle points.
• fast_local_convergence::Bool = true: if true, we scale the regularization parameter σ by the norm of the current gradient on very successful iterations (using γ3 * min(σ, norm(gx))), which accelerates local convergence near a minimizer.
• npc_handler::Symbol = :ag: the nonpositivecurve handling strategy.
  • :ag: run line-search along NPC with Armijo-Goldstein conditions.
  • :sigma: increase the regularization parameter σ.
  • :prev: use the subsolver's last iterate before negative curvature was detected, sk = sk^(t-1). If it is detected at the first iteration, the Krylov subsolver returns -∇f instead; σ is not increased in either case.
  • :cp: set s_k to Cauchy point.
  • Note: For exact direct subsolvers (e.g., MA57R2NSubsolver), an exact solution to an indefinite shifted system cannot simultaneously be a strict descent direction. If the factorized matrix is singular or indefinite, the solver automatically rejects the step and increases σ, bypassing this handler.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = R2N(nlp)

Passing a pre-built HSL subsolver with input-size guards (e.g. to bail out on very large problems instead of paying HSL's eager symbolic analysis):

using JSOSolvers, ADNLPModels, HSL
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
sub = MA57R2NSubsolver(nlp; max_nvar = 100_000, max_nnzh = 5_000_000)
stats = R2N(nlp; subsolver = sub)

Same pattern works for MA97R2NSubsolver. If a guard trips, stats.status == :exception.

Dense Hessians are only rejected above dense_max_nvar (default 12_000), so a dense n = 5_000 problem is still factorized. Lower it to reject dense problems earlier, or raise it to allow bigger dense ones:

sub = MA57R2NSubsolver(nlp; dense_max_nvar = 20_000)

`R2Solver` is deprecated, please check the documentation of `R2`.

TRONLSParameterSet{T} <: AbstractParameterSet

This structure designed for tron regroups the following parameters:

• μ₀: algorithm parameter in (0, 0.5).
• μ₁: algorithm parameter in (0, +∞).
• σ: algorithm parameter in (1, +∞).

An additional constructor is

TRONLSParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• μ₀::T = T(1 / 100)
• μ₁::T = T(1)
• σ::T = T(10)

TRONParameterSet{T} <: AbstractParameterSet

This structure designed for tron regroups the following parameters:

• μ₀::T: algorithm parameter in (0, 0.5).
• μ₁::T: algorithm parameter in (0, +∞).
• σ::T: algorithm parameter in (1, +∞).

An additional constructor is

TRONParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• μ₀::T = T(1 / 100)
• μ₁::T = T(1)
• σ::T = T(10)

TRUNKLSParameterSet <: AbstractParameterSet

This structure designed for tron regroups the following parameters:

• bk_max: algorithm parameter.
• monotone: algorithm parameter.
• nm_itmax: algorithm parameter.

An additional constructor is

TRUNKLSParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• bk_max::Int = 10
• monotone::Bool = true
• nm_itmax::Int = 25

TRUNKParameterSet <: AbstractParameterSet

This structure designed for tron regroups the following parameters:

• bk_max: algorithm parameter.
• monotone: algorithm parameter.
• nm_itmax: algorithm parameter.

An additional constructor is

TRUNKParameterSet(nlp: kwargs...)

where the kwargs are the parameters above.

Default values are:

• bk_max::Int = 10
• monotone::Bool = true
• nm_itmax::Int = 25

tron(nlp; kwargs...)

A pure Julia implementation of a trust-region solver for bound-constrained optimization:

    min f(x)    s.t.    ℓ ≦ x ≦ u

For advanced usage, first define a TronSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = TronSolver(nlp; kwargs...)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• μ₀::T = T(1 / 100): algorithm parameter, see TRONParameterSet.
• μ₁::T = T(1): algorithm parameter, see TRONParameterSet.
• σ::T = T(10): algorithm parameter, see TRONParameterSet.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• max_cgiter::Int = 50: subproblem's iteration limit.
• use_only_objgrad::Bool = false: If true, the algorithm uses only the function objgrad instead of obj and grad.
• cgtol::T = T(0.1): subproblem tolerance.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖x - Proj(x - ∇f(xᵏ))‖ ≤ atol + rtol * ‖∇f(x⁰)‖. Proj denotes here the projection over the bounds.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

The keyword arguments of TronSolver are passed to the TRONTrustRegion constructor.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

TRON is described in

Chih-Jen Lin and Jorge J. Moré, *Newton's Method for Large Bound-Constrained
Optimization Problems*, SIAM J. Optim., 9(4), 1100–1127, 1999.
DOI: 10.1137/S1052623498345075

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x), ones(3), zeros(3), 2 * ones(3));
stats = tron(nlp)

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x), ones(3), zeros(3), 2 * ones(3));
solver = TronSolver(nlp);
stats = solve!(solver, nlp)

tron(nls; kwargs...)

A pure Julia implementation of a trust-region solver for bound-constrained nonlinear least-squares problems:

min ½‖F(x)‖²    s.t.    ℓ ≦ x ≦ u

For advanced usage, first define a TronSolverNLS to preallocate the memory used in the algorithm, and then call solve!: solver = TronSolverNLS(nls, subsolver::Symbol=:lsmr; kwargs...) solve!(solver, nls; kwargs...)

Arguments

• nls::AbstractNLSModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• subsolver::Symbol = :lsmr: Krylov.jl method used as subproblem solver, see JSOSolvers.tronls_allowed_subsolvers for a list.
• μ₀::T = T(1 / 100): algorithm parameter, see TRONLSParameterSet.
• μ₁::T = T(1): algorithm parameter, see TRONLSParameterSet.
• σ::T = T(10): algorithm parameter, see TRONLSParameterSet.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• max_cgiter::Int = 50: subproblem iteration limit.
• cgtol::T = T(0.1): subproblem tolerance.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖x - Proj(x - ∇f(xᵏ))‖ ≤ atol + rtol * ‖∇f(x⁰)‖. Proj denotes here the projection over the bounds.
• Fatol::T = √eps(T): absolute tolerance on the residual.
• Frtol::T = eps(T): relative tolerance on the residual, the algorithm stops when ‖F(xᵏ)‖ ≤ Fatol + Frtol * ‖F(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

The keyword arguments of TronSolverNLS are passed to the TRONTrustRegion constructor.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

This is an adaptation for bound-constrained nonlinear least-squares problems of the TRON method described in

Chih-Jen Lin and Jorge J. Moré, *Newton's Method for Large Bound-Constrained
Optimization Problems*, SIAM J. Optim., 9(4), 1100–1127, 1999.
DOI: 10.1137/S1052623498345075

Examples

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2, zeros(2), 0.5 * ones(2))
stats = tron(nls)

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2, zeros(2), 0.5 * ones(2))
solver = TronSolverNLS(nls)
stats = solve!(solver, nls)

trunk(nlp; kwargs...)

A trust-region solver for unconstrained optimization using exact second derivatives.

For advanced usage, first define a TrunkSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = TrunkSolver(nlp, subsolver::Symbol = :cg)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• subsolver_logger::AbstractLogger = NullLogger(): subproblem's logger.
• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• bk_max::Int = 10: algorithm parameter, see TRUNKParameterSet.
• monotone::Bool = true: algorithm parameter, see TRUNKParameterSet.
• nm_itmax::Int = 25: algorithm parameter, see TRUNKParameterSet.
• verbose::Int = 0: if > 0, display iteration information every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.
• M: linear operator that models a Hermitian positive-definite matrix of size n; passed to Krylov subsolvers.

Output

The returned value is a GenericExecutionStats, see SolverCore.jl.

References

This implementation follows the description given in

A. R. Conn, N. I. M. Gould, and Ph. L. Toint,
Trust-Region Methods, volume 1 of MPS/SIAM Series on Optimization.
SIAM, Philadelphia, USA, 2000.
DOI: 10.1137/1.9780898719857

The main algorithm follows the basic trust-region method described in Section 6. The backtracking linesearch follows Section 10.3.2. The nonmonotone strategy follows Section 10.1.3, Algorithm 10.1.2.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = trunk(nlp)

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = TrunkSolver(nlp)
stats = solve!(solver, nlp)

trunk(nls; kwargs...)

A pure Julia implementation of a trust-region solver for nonlinear least-squares problems:

min ½‖F(x)‖²

For advanced usage, first define a TrunkSolverNLS to preallocate the memory used in the algorithm, and then call solve!:

solver = TrunkSolverNLS(nls, subsolver::Symbol = :lsmr)
solve!(solver, nls; kwargs...)

Arguments

• nls::AbstractNLSModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• Fatol::T = √eps(T): absolute tolerance on the residual.
• Frtol::T = eps(T): relative tolerance on the residual, the algorithm stops when ‖F(xᵏ)‖ ≤ Fatol + Frtol * ‖F(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• bk_max::Int = 10: algorithm parameter, see TRUNKLSParameterSet.
• monotone::Bool = true: algorithm parameter, see TRUNKLSParameterSet.
• nm_itmax::Int = 25: algorithm parameter, see TRUNKLSParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

See JSOSolvers.trunkls_allowed_subsolvers for a list of available Krylov solvers.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

This implementation follows the description given in

A. R. Conn, N. I. M. Gould, and Ph. L. Toint,
Trust-Region Methods, volume 1 of MPS/SIAM Series on Optimization.
SIAM, Philadelphia, USA, 2000.
DOI: 10.1137/1.9780898719857

The main algorithm follows the basic trust-region method described in Section 6. The backtracking linesearch follows Section 10.3.2. The nonmonotone strategy follows Section 10.1.3, Algorithm 10.1.2.

Examples

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2)
stats = trunk(nls)

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2)
solver = TrunkSolverNLS(nls)
stats = solve!(solver, nls)

MA97R2NSubsolver(nlp; fill_ratio = 0.5, dense_max_nvar = 12_000,
                 max_nvar = typemax(Int), max_nnzh = typemax(Int))
MA57R2NSubsolver(nlp; fill_ratio = 0.5, dense_max_nvar = 12_000,
                 max_nvar = typemax(Int), max_nnzh = typemax(Int))

Direct sparse subsolvers for R2N backed by HSL's MA97 and MA57. They are only available after using HSL, and running them requires a licensed HSL_jll.

Before touching HSL, a cheap input-size and density guard is applied. If any of

• nlp.meta.nvar > max_nvar,
• nlp.meta.nnzh > max_nnzh (nnzh counts stored lower-triangular Hessian nonzeros), or
• nnzh > fill_ratio * n(n+1)/2 when n > dense_max_nvar

is true, the constructor returns an "unsupported" placeholder: no HSL object is allocated, no symbolic analysis is run, and is_unsupported returns true. R2N then short-circuits to status :exception. Use this to avoid HSL's eager symbolic analysis on problems that are too big to factorize (e.g. n ≈ 120_000).

Density guard and problem size

A dense Hessian is cheap enough to factorize at moderate n, so the fill_ratio density test only applies once n > dense_max_nvar: any problem with n ≤ dense_max_nvar is handed to HSL no matter how dense it is. With the default dense_max_nvar = 12_000, a dense n = 5_000 problem still runs through HSL.

Set dense_max_nvar to the largest n for which you accept factorizing a dense Hessian (work grows like n³, storage like n²). dense_max_nvar = 0 restores the "reject every dense Hessian" behaviour, while max_nvar/max_nnzh stay hard caps that apply regardless of density.

MA97R2NSubsolver(nlp; fill_ratio = 0.5, dense_max_nvar = 12_000,
                 max_nvar = typemax(Int), max_nnzh = typemax(Int))
MA57R2NSubsolver(nlp; fill_ratio = 0.5, dense_max_nvar = 12_000,
                 max_nvar = typemax(Int), max_nnzh = typemax(Int))

Direct sparse subsolvers for R2N backed by HSL's MA97 and MA57. They are only available after using HSL, and running them requires a licensed HSL_jll.

Before touching HSL, a cheap input-size and density guard is applied. If any of

• nlp.meta.nvar > max_nvar,
• nlp.meta.nnzh > max_nnzh (nnzh counts stored lower-triangular Hessian nonzeros), or
• nnzh > fill_ratio * n(n+1)/2 when n > dense_max_nvar

is true, the constructor returns an "unsupported" placeholder: no HSL object is allocated, no symbolic analysis is run, and is_unsupported returns true. R2N then short-circuits to status :exception. Use this to avoid HSL's eager symbolic analysis on problems that are too big to factorize (e.g. n ≈ 120_000).

Density guard and problem size

A dense Hessian is cheap enough to factorize at moderate n, so the fill_ratio density test only applies once n > dense_max_nvar: any problem with n ≤ dense_max_nvar is handed to HSL no matter how dense it is. With the default dense_max_nvar = 12_000, a dense n = 5_000 problem still runs through HSL.

Set dense_max_nvar to the largest n for which you accept factorizing a dense Hessian (work grows like n³, storage like n²). dense_max_nvar = 0 restores the "reject every dense Hessian" behaviour, while max_nvar/max_nnzh stay hard caps that apply regardless of density.

QRMumpsSubsolver(nls; fill_ratio = 0.5, dense_max_entries = 12_000^2,
                 max_nvar = typemax(Int), max_nnzj = typemax(Int))

QRMumpsR2NLSSubsolver constructor. Same guards and keyword semantics.

fo(nlp; kwargs...)
R2(nlp; kwargs...)
TR(nlp; kwargs...)

A First-Order (FO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FoSolver(nlp)
solve!(solver, nlp; kwargs...)

R2 and TR runs fo with the dedicated step_backend keyword argument.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• η1 = eps(T)^(1 // 4): algorithm parameter, see FOMOParameterSet.
• η2 = T(95/100): algorithm parameter, see FOMOParameterSet.
• γ1 = T(1/2): algorithm parameter, see FOMOParameterSet.
• γ2 = T(2): algorithm parameter, see FOMOParameterSet.
• αmax = 1/eps(T): algorithm parameter, see FOMOParameterSet.
• α0 = T(0): algorithm parameter, see FOMOParameterSet.
• max_eval::Int = -1: maximum number of evaluation of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• M = 1 : algorithm parameter, see FOMOParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): algorithm parameter, see FOMOParameterSet.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fo(nlp) # run with step_backend = r2_step(), equivalent to R2(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

R2N(nlp; kwargs...)

A second-order quadratic regularization method for unconstrained optimization (with shifted L-BFGS or shifted Hessian operator).

min f(x)

For advanced usage, first define a R2NSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = R2NSolver(nlp)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• η1::T = eps(T)^(1/4): step acceptance parameter, see R2NParameterSet.
• η2::T = T(0.95): step acceptance parameter, see R2NParameterSet.
• θ1::T = T(0.9): Cauchy step parameter, see R2NParameterSet.
• θ2::T = eps(T)^(-1): Cauchy step parameter, see R2NParameterSet.
• γ1::T = T(1.5): regularization update parameter, see R2NParameterSet.
• γ2::T = T(2.5): regularization update parameter, see R2NParameterSet.
• γ3::T = T(0.5): regularization update parameter, see R2NParameterSet.
• δ1::T = T(0.5): Cauchy point calculation parameter, see R2NParameterSet.
• σmin::T = eps(T): minimum step parameter, see R2NParameterSet.
• non_mono_size::Int = 1: the size of the non-monotone behaviour. If > 1, the algorithm will use a non-monotone strategy to accept steps.
• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• max_eval::Int = -1: maximum number of evaluations of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver = CGR2NSubsolver: the subproblem solver type or instance.
  • Note: HSL guards fill_ratio, dense_max_nvar, max_nvar, and max_nnzh must be passed to a pre-built subsolver, e.g. subsolver = MA57R2NSubsolver(nlp; max_nvar = 100_000, max_nnzh = 5_000_000). max_nvar is compared against nlp.meta.nvar and max_nnzh against nlp.meta.nnzh (the number of stored lower-triangular Hessian nonzeros). Both default to typemax(Int) (no additional limit). The fill_ratio density guard only applies once nvar > dense_max_nvar (default 12_000), so dense problems up to that size are still factorized by HSL. If any guard trips, HSL allocation and symbolic analysis are skipped and R2N returns status :exception. These are cheap input-size checks, not bounds on factorization memory or runtime.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver if KrylovWorkspace type is selected.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• scp_flag::Bool = false: if true, we compare the norm of the calculate step with θ2 * norm(scp), each iteration, selecting the smaller step.
• always_accept_npc_ag::Bool = false: if true, we skip the computation of the reduction ratio ρ for Goldstein steps taken along directions of negative curvature, unconditionally accepting them as successful steps to aggressively escape saddle points.
• fast_local_convergence::Bool = true: if true, we scale the regularization parameter σ by the norm of the current gradient on very successful iterations (using γ3 * min(σ, norm(gx))), which accelerates local convergence near a minimizer.
• npc_handler::Symbol = :ag: the nonpositivecurve handling strategy.
  • :ag: run line-search along NPC with Armijo-Goldstein conditions.
  • :sigma: increase the regularization parameter σ.
  • :prev: use the subsolver's last iterate before negative curvature was detected, sk = sk^(t-1). If it is detected at the first iteration, the Krylov subsolver returns -∇f instead; σ is not increased in either case.
  • :cp: set s_k to Cauchy point.
  • Note: For exact direct subsolvers (e.g., MA57R2NSubsolver), an exact solution to an indefinite shifted system cannot simultaneously be a strict descent direction. If the factorized matrix is singular or indefinite, the solver automatically rejects the step and increases σ, bypassing this handler.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = R2N(nlp)

Passing a pre-built HSL subsolver with input-size guards (e.g. to bail out on very large problems instead of paying HSL's eager symbolic analysis):

using JSOSolvers, ADNLPModels, HSL
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
sub = MA57R2NSubsolver(nlp; max_nvar = 100_000, max_nnzh = 5_000_000)
stats = R2N(nlp; subsolver = sub)

Same pattern works for MA97R2NSubsolver. If a guard trips, stats.status == :exception.

Dense Hessians are only rejected above dense_max_nvar (default 12_000), so a dense n = 5_000 problem is still factorized. Lower it to reject dense problems earlier, or raise it to allow bigger dense ones:

sub = MA57R2NSubsolver(nlp; dense_max_nvar = 20_000)

fo(nlp; kwargs...)
R2(nlp; kwargs...)
TR(nlp; kwargs...)

A First-Order (FO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FoSolver(nlp)
solve!(solver, nlp; kwargs...)

R2 and TR runs fo with the dedicated step_backend keyword argument.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• η1 = eps(T)^(1 // 4): algorithm parameter, see FOMOParameterSet.
• η2 = T(95/100): algorithm parameter, see FOMOParameterSet.
• γ1 = T(1/2): algorithm parameter, see FOMOParameterSet.
• γ2 = T(2): algorithm parameter, see FOMOParameterSet.
• αmax = 1/eps(T): algorithm parameter, see FOMOParameterSet.
• α0 = T(0): algorithm parameter, see FOMOParameterSet.
• max_eval::Int = -1: maximum number of evaluation of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• M = 1 : algorithm parameter, see FOMOParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): algorithm parameter, see FOMOParameterSet.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fo(nlp) # run with step_backend = r2_step(), equivalent to R2(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

α, s = cauchy!(x, H, g, Δ, ℓ, u, s, Hs; μ₀ = 1e-2, μ₁ = 1.0, σ = 10.0)

Computes a Cauchy step s = P(x - α g) - x for

min  q(s) = ¹/₂sᵀHs + gᵀs     s.t.    ‖s‖ ≦ μ₁Δ,  ℓ ≦ x + s ≦ u,

with the sufficient decrease condition

q(s) ≦ μ₀sᵀg.

α, s = cauchy_ls!(x, A, Fx, g, Δ, ℓ, u, s, As; μ₀ = 1e-2, μ₁ = 1.0, σ = 10.0)

Computes a Cauchy step s = P(x - α g) - x for

min  q(s) = ½‖As + Fx‖² - ½‖Fx‖²     s.t.    ‖s‖ ≦ μ₁Δ,  ℓ ≦ x + s ≦ u,

with the sufficient decrease condition

q(s) ≦ μ₀gᵀs,

where g = AᵀFx.

default_callback_quasi_newton(nlp, solver, stats)

A default callback for solvers to update the Hessian approximation in quasi-Newton models. If a user calls a solver with a quasi-Newton model, this will be the default callback. See https://jso.dev/JSOSolvers.jl/stable/#Callbacks

dense_hessian_entries(n)

Number of stored entries of a dense symmetric n × n Hessian, n(n+1)/2.

dense_jacobian_entries(m, n)

Number of entries of a dense m × n Jacobian, m * n.

finalize_subsolver!(subsolver::AbstractSubsolver)

Explicitly free any C/Fortran memory allocated by the subsolver to prevent leaks.

find_beta(m, mdot∇f, norm_∇f, μk, fk, max_obj_mem, β, θ1, θ2)

Compute βmax which saturates the contribution of the momentum term to the gradient. βmax is computed such that the two gradient-related conditions (first one is relaxed in the nonmonotone case) are ensured:

1. (1-βmax) * ‖∇f(xk)‖² + βmax * ∇f(xk)ᵀm + (maxobjmem - fk)/μk ≥ θ1 * ‖∇f(xk)‖²
2. ‖∇f(xk)‖ ≥ θ2 * ‖(1-βmax) * ∇f(xk) .+ βmax .* m‖

with m the momentum term and mdot∇f = ∇f(xk)ᵀm, fk the model at s=0, max_obj_mem the largest objective value over the last M successful iterations.

fo(nlp; kwargs...)
R2(nlp; kwargs...)
TR(nlp; kwargs...)

A First-Order (FO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FoSolver(nlp)
solve!(solver, nlp; kwargs...)

R2 and TR runs fo with the dedicated step_backend keyword argument.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• η1 = eps(T)^(1 // 4): algorithm parameter, see FOMOParameterSet.
• η2 = T(95/100): algorithm parameter, see FOMOParameterSet.
• γ1 = T(1/2): algorithm parameter, see FOMOParameterSet.
• γ2 = T(2): algorithm parameter, see FOMOParameterSet.
• αmax = 1/eps(T): algorithm parameter, see FOMOParameterSet.
• α0 = T(0): algorithm parameter, see FOMOParameterSet.
• max_eval::Int = -1: maximum number of evaluation of the objective function.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• M = 1 : algorithm parameter, see FOMOParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): algorithm parameter, see FOMOParameterSet.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fo(nlp) # run with step_backend = r2_step(), equivalent to R2(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

fomo(nlp; kwargs...)

A First-Order with MOmentum (FOMO) model-based method for unconstrained optimization. Supports quadratic regularization and trust region method with linear model.

Algorithm description

The step is computed along d = - (1-βmax) .* ∇f(xk) - βmax .* mk with mk the memory of past gradients (initialized at 0), and updated at each successful iteration as mk .= ∇f(xk) .* (1 - βmax) .+ mk .* βmax and βmax ∈ [0,β] chosen as to ensure d is gradient-related, i.e., the following 2 conditions are satisfied: (1-βmax) .* ∇f(xk) + βmax .* ∇f(xk)ᵀmk ≥ θ1 * ‖∇f(xk)‖² (1) ‖∇f(xk)‖ ≥ θ2 * ‖(1-βmax) . ∇f(xk) + βmax . mk‖ (2) In the nonmonotone case, (1) rewrites (1-βmax) .* ∇f(xk) + βmax .* ∇f(xk)ᵀmk + (fm - fk)/μk ≥ θ1 * ‖∇f(xk)‖², with fm the largest objective value over the last M successful iterations, and fk = f(xk).

Advanced usage

For advanced usage, first define a FomoSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = FomoSolver(nlp)
solve!(solver, nlp; kwargs...)

No momentum: if the user does not whish to use momentum (β = 0), it is recommended to use the memory-optimized fo method.

Arguments

• nlp::AbstractNLPModel{T, V} is the model to solve, see NLPModels.jl.

Keyword arguments

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance: algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• η1 = eps(T)^(1 // 4), η2 = T(95/100): step acceptance parameters.
• γ1 = T(1/2), γ2 = T(2): regularization update parameters.
• γ3 = T(1/2) : momentum factor βmax update parameter in case of unsuccessful iteration.
• αmax = 1/eps(T): maximum step parameter for fomo algorithm.
• α0 = T(0): initial value of the step parameter (capped by αmax); if 0, it is chosen from the initial gradient norm.
• max_eval::Int = -1: maximum number of objective evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• β = T(9/10) ∈ [0,1): target decay rate for the momentum.
• θ1 = T(1/10): momentum contribution parameter for convergence condition (1).
• θ2 = eps(T)^(1/3): momentum contribution parameter for convergence condition (2).
• M = 1 : requires objective decrease over the M last iterates (nonmonotone context). M=1 implies monotone behaviour.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• step_backend = r2_step(): step computation mode. Options are r2_step() for quadratic regulation step and tr_step() for first-order trust-region.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

Examples

fomo

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = fomo(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = FomoSolver(nlp);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

get_inertia(subsolver)

Return (numneg, numzero) eigenvalues of the underlying operator. Returns (-1, -1) if unknown or not applicable.

get_jacobian(subsolver)

Return the operator/matrix J for outer loop calculations in R2NLS.

get_jacobian_norm(subsolver)

Return an estimate or upper bound of ‖J‖, the norm of the Jacobian returned by get_jacobian. Used by R2NLS for the Cauchy point.

get_model_hessian(subsolver)

Return the model Hessian H (operator or matrix) used by R2N in the outer loop (curvature, Cauchy point).

get_model_hessian_norm(subsolver)

Return an estimate or upper bound of ‖H‖, the norm of the model Hessian used by R2N. Used for the Cauchy point.

get_npc_direction(subsolver)

Return a direction of negative curvature. Raises an error if none was found or if the subsolver does not support it.

has_npc_direction(subsolver)

Return true if a direction of negative curvature was found during the last solve, false otherwise.

init_alpha(norm_∇fk::T, ::r2_step)
init_alpha(norm_∇fk::T, ::tr_step)

Initialize α step size parameter. Ensure first step is the same for quadratic regularization and trust region methods.

initialize!(subsolver::AbstractSubsolver, args...)

Perform initial setup for the subsolver (e.g., analyze sparsity, allocate workspace). This is typically called once at the start of the optimization.

is_unsupported(subsolver)

Return true if the subsolver cannot be used for the current problem (e.g. the Hessian is too dense for a direct sparse factorization). When true, the outer solver should skip solving and return with status :exception. Defaults to false.

lbfgs(nlp; kwargs...)

An implementation of a limited memory BFGS line-search method for unconstrained minimization.

For advanced usage, first define a LBFGSSolver to preallocate the memory used in the algorithm, and then call solve!.

solver = LBFGSSolver(nlp; mem::Int = 5)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• mem::Int = 5: algorithm parameter, see LBFGSParameterSet.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• τ₁::T = T(0.9999): algorithm parameter, see LBFGSParameterSet.
• bk_max:: Int = 25: algorithm parameter, see LBFGSParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• verbose_subsolver::Int = 0: if > 0, display iteration information every verbose_subsolver iteration of the subsolver.

Output

The returned value is a GenericExecutionStats, see SolverCore.jl.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3));
stats = lbfgs(nlp)

# output

"Execution stats: first-order stationary"

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3));
solver = LBFGSSolver(nlp; mem = 5);
stats = solve!(solver, nlp)

# output

"Execution stats: first-order stationary"

normM!(n, x, M, z)

Weighted norm of x with respect to M, i.e., z = sqrt(x' * M * x). Uses z as workspace.

projected_gauss_newton!(solver, x, A, Fx, Δ, gctol, s, max_cgiter, ℓ, u; max_cgiter = 50, max_time = Inf, subsolver_verbose = 0)

Compute an approximate solution d for

min q(d) = ½‖Ad + Fx‖² - ½‖Fx‖²     s.t.    ℓ ≦ x + d ≦ u,  ‖d‖ ≦ Δ

starting from s. The steps are computed using the conjugate gradient method projected on the active bounds.

s = projected_line_search!(x, H, g, d, ℓ, u, Hs, μ₀)

Performs a projected line search, searching for a step size t such that

0.5sᵀHs + sᵀg ≦ μ₀sᵀg,

where s = P(x + t * d) - x, while remaining on the same face as x + d. Backtracking is performed from t = 1.0. x is updated in place.

s = projected_line_search_ls!(x, A, g, d, ℓ, u, As, s; μ₀ = 1e-2)

Performs a projected line search, searching for a step size t such that

½‖As + Fx‖² ≤ ½‖Fx‖² + μ₀FxᵀAs

where s = P(x + t * d) - x, while remaining on the same face as x + d. Backtracking is performed from t = 1.0. x is updated in place.

projected_newton!(solver, x, H, g, Δ, cgtol, ℓ, u, s, Hs; max_time = Inf, max_cgiter = 50, subsolver_verbose = 0)

Compute an approximate solution d for

min q(d) = ¹/₂dᵀHs + dᵀg    s.t.    ℓ ≦ x + d ≦ u,  ‖d‖ ≦ Δ

starting from s. The steps are computed using the conjugate gradient method projected on the active bounds.

reset_subsolver!(subsolver::AbstractSubsolver, nlp, x)

Reset the subsolver state. Default behavior is to finalize and re-initialize.

step_mult(α::T, norm_∇fk::T, ::r2_step)
step_mult(α::T, norm_∇fk::T, ::tr_step)

Compute step size multiplier: α for quadratic regularization(::r2 and ::R2og) and α/norm_∇fk for trust region (::tr).

tron(nls; kwargs...)

A pure Julia implementation of a trust-region solver for bound-constrained nonlinear least-squares problems:

min ½‖F(x)‖²    s.t.    ℓ ≦ x ≦ u

For advanced usage, first define a TronSolverNLS to preallocate the memory used in the algorithm, and then call solve!: solver = TronSolverNLS(nls, subsolver::Symbol=:lsmr; kwargs...) solve!(solver, nls; kwargs...)

Arguments

• nls::AbstractNLSModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• subsolver::Symbol = :lsmr: Krylov.jl method used as subproblem solver, see JSOSolvers.tronls_allowed_subsolvers for a list.
• μ₀::T = T(1 / 100): algorithm parameter, see TRONLSParameterSet.
• μ₁::T = T(1): algorithm parameter, see TRONLSParameterSet.
• σ::T = T(10): algorithm parameter, see TRONLSParameterSet.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• max_cgiter::Int = 50: subproblem iteration limit.
• cgtol::T = T(0.1): subproblem tolerance.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖x - Proj(x - ∇f(xᵏ))‖ ≤ atol + rtol * ‖∇f(x⁰)‖. Proj denotes here the projection over the bounds.
• Fatol::T = √eps(T): absolute tolerance on the residual.
• Frtol::T = eps(T): relative tolerance on the residual, the algorithm stops when ‖F(xᵏ)‖ ≤ Fatol + Frtol * ‖F(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

The keyword arguments of TronSolverNLS are passed to the TRONTrustRegion constructor.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

This is an adaptation for bound-constrained nonlinear least-squares problems of the TRON method described in

Chih-Jen Lin and Jorge J. Moré, *Newton's Method for Large Bound-Constrained
Optimization Problems*, SIAM J. Optim., 9(4), 1100–1127, 1999.
DOI: 10.1137/S1052623498345075

Examples

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2, zeros(2), 0.5 * ones(2))
stats = tron(nls)

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2, zeros(2), 0.5 * ones(2))
solver = TronSolverNLS(nls)
stats = solve!(solver, nls)

tron(nlp; kwargs...)

A pure Julia implementation of a trust-region solver for bound-constrained optimization:

    min f(x)    s.t.    ℓ ≦ x ≦ u

For advanced usage, first define a TronSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = TronSolver(nlp; kwargs...)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• μ₀::T = T(1 / 100): algorithm parameter, see TRONParameterSet.
• μ₁::T = T(1): algorithm parameter, see TRONParameterSet.
• σ::T = T(10): algorithm parameter, see TRONParameterSet.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• max_cgiter::Int = 50: subproblem's iteration limit.
• use_only_objgrad::Bool = false: If true, the algorithm uses only the function objgrad instead of obj and grad.
• cgtol::T = T(0.1): subproblem tolerance.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖x - Proj(x - ∇f(xᵏ))‖ ≤ atol + rtol * ‖∇f(x⁰)‖. Proj denotes here the projection over the bounds.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

The keyword arguments of TronSolver are passed to the TRONTrustRegion constructor.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

TRON is described in

Chih-Jen Lin and Jorge J. Moré, *Newton's Method for Large Bound-Constrained
Optimization Problems*, SIAM J. Optim., 9(4), 1100–1127, 1999.
DOI: 10.1137/S1052623498345075

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x), ones(3), zeros(3), 2 * ones(3));
stats = tron(nlp)

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x), ones(3), zeros(3), 2 * ones(3));
solver = TronSolver(nlp);
stats = solve!(solver, nlp)

trunk(nls; kwargs...)

A pure Julia implementation of a trust-region solver for nonlinear least-squares problems:

min ½‖F(x)‖²

For advanced usage, first define a TrunkSolverNLS to preallocate the memory used in the algorithm, and then call solve!:

solver = TrunkSolverNLS(nls, subsolver::Symbol = :lsmr)
solve!(solver, nls; kwargs...)

Arguments

• nls::AbstractNLSModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• Fatol::T = √eps(T): absolute tolerance on the residual.
• Frtol::T = eps(T): relative tolerance on the residual, the algorithm stops when ‖F(xᵏ)‖ ≤ Fatol + Frtol * ‖F(x⁰)‖.
• callback: function called at each iteration, see Callback section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• bk_max::Int = 10: algorithm parameter, see TRUNKLSParameterSet.
• monotone::Bool = true: algorithm parameter, see TRUNKLSParameterSet.
• nm_itmax::Int = 25: algorithm parameter, see TRUNKLSParameterSet.
• verbose::Int = 0: if > 0, display iteration details every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.

See JSOSolvers.trunkls_allowed_subsolvers for a list of available Krylov solvers.

Output

The value returned is a GenericExecutionStats, see SolverCore.jl.

References

This implementation follows the description given in

A. R. Conn, N. I. M. Gould, and Ph. L. Toint,
Trust-Region Methods, volume 1 of MPS/SIAM Series on Optimization.
SIAM, Philadelphia, USA, 2000.
DOI: 10.1137/1.9780898719857

The main algorithm follows the basic trust-region method described in Section 6. The backtracking linesearch follows Section 10.3.2. The nonmonotone strategy follows Section 10.1.3, Algorithm 10.1.2.

Examples

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2)
stats = trunk(nls)

using JSOSolvers, ADNLPModels
F(x) = [x[1] - 1.0; 10 * (x[2] - x[1]^2)]
x0 = [-1.2; 1.0]
nls = ADNLSModel(F, x0, 2)
solver = TrunkSolverNLS(nls)
stats = solve!(solver, nls)

trunk(nlp; kwargs...)

A trust-region solver for unconstrained optimization using exact second derivatives.

For advanced usage, first define a TrunkSolver to preallocate the memory used in the algorithm, and then call solve!:

solver = TrunkSolver(nlp, subsolver::Symbol = :cg)
solve!(solver, nlp; kwargs...)

Arguments

• nlp::AbstractNLPModel{T, V} represents the model to solve, see NLPModels.jl.

The keyword arguments may include

• subsolver_logger::AbstractLogger = NullLogger(): subproblem's logger.
• x::V = nlp.meta.x0: the initial guess.
• atol::T = √eps(T): absolute tolerance.
• rtol::T = √eps(T): relative tolerance, the algorithm stops when ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖.
• callback: function called at each iteration, see Callbacks section.
• callback_quasi_newton: function called at each iteration, specifically to update the Hessian approximation of quasi-Newton models, see Callbacks section.
• max_eval::Int = -1: maximum number of objective function evaluations.
• max_time::Float64 = 30.0: maximum time limit in seconds.
• max_iter::Int = typemax(Int): maximum number of iterations.
• bk_max::Int = 10: algorithm parameter, see TRUNKParameterSet.
• monotone::Bool = true: algorithm parameter, see TRUNKParameterSet.
• nm_itmax::Int = 25: algorithm parameter, see TRUNKParameterSet.
• verbose::Int = 0: if > 0, display iteration information every verbose iteration.
• subsolver_verbose::Int = 0: if > 0, display iteration information every subsolver_verbose iteration of the subsolver.
• M: linear operator that models a Hermitian positive-definite matrix of size n; passed to Krylov subsolvers.

Output

The returned value is a GenericExecutionStats, see SolverCore.jl.

References

This implementation follows the description given in

A. R. Conn, N. I. M. Gould, and Ph. L. Toint,
Trust-Region Methods, volume 1 of MPS/SIAM Series on Optimization.
SIAM, Philadelphia, USA, 2000.
DOI: 10.1137/1.9780898719857

The main algorithm follows the basic trust-region method described in Section 6. The backtracking linesearch follows Section 10.3.2. The nonmonotone strategy follows Section 10.1.3, Algorithm 10.1.2.

Examples

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
stats = trunk(nlp)

using JSOSolvers, ADNLPModels
nlp = ADNLPModel(x -> sum(x.^2), ones(3))
solver = TrunkSolver(nlp)
stats = solve!(solver, nlp)

update_subsolver!(subsolver::AbstractSubsolver, nlp, x)

Update the internal representation of the subsolver at point x. For R2N solvers, this typically updates the Hessian or Operator. For R2NLS solvers, this updates the Jacobian.