Tutorial

The package SolverTest provides a set a functions to test the basic features of a JSO-compliant solver. It is assumed that:

  1. the tested solver accepts an AbstractNLPModel as input, and;
  2. returns a GenericExecutionStats as output;
  3. The solver function should accepts atol and rtol as keyword arguments.

As shown in the examples below, the package also provide multiprecision tests for compatible solvers.

Unconstrained Optimization

The function unconstrained_nlp_set returns a list of optimization problems in the ADNLPModel format.

Secondly, the function unconstrained_nlp will test the behavior of the JSO-compliant function solver on the test set. More precisely, it will test that primal and dual feasibility at the solution (stats.solution) computed with kkt_checker and that stats.dual_feas are relatively small, i.e. ‖∇f(xᵏ)‖ ≤ atol + rtol * ‖∇f(x⁰)‖ where atol = rtol = 1e-6. It will also be tested that the final status stats.status is :first_order. The problems are sufficiently simple that we expect the solver to solve them successfully.

using SolverTest
using Test
using JSOSolvers # export solver trunk
@testset "Unconstrained solvers" begin
  unconstrained_nlp(trunk)
  multiprecision_nlp(trunk, :unc)
end
Test.DefaultTestSet("Unconstrained solvers", Any[Test.DefaultTestSet("Problem (x₁ - 1)² + 4(x₂ - 1)²", Any[], 4, false, false, true, 1.748902411867771e9, 1.748902425716217e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Diagonal quadratic", Any[], 4, false, false, true, 1.748902425716247e9, 1.748902426907424e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Tridiagonal quadratic", Any[], 4, false, false, true, 1.748902426907453e9, 1.748902428236496e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Rosenbrock", Any[], 4, false, false, true, 1.748902428236527e9, 1.748902428890416e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Extended Rosenbrock", Any[], 4, false, false, true, 1.748902428890445e9, 1.748902429690295e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Precision Float16 for ptype unc", Any[], 7, false, false, true, 1.74890247430603e9, 1.748902497021374e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype unc", Any[], 7, false, false, true, 1.748902497021419e9, 1.748902518181132e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype unc", Any[], 7, false, false, true, 1.748902518181171e9, 1.748902519029141e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype unc", Any[], 7, false, false, true, 1.748902519029188e9, 1.748902539104055e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl")], 0, false, false, true, 1.748902397717668e9, 1.748902539110529e9, false, "none")

Similarly, unconstrained_nls_set and unconstrained_nls are preparing the same tests for nonlinear least squares.

using SolverTest
using Test
using JSOSolvers # export solver trunk
@testset "Unconstrained solvers" begin
  unconstrained_nls(trunk)
  multiprecision_nls(trunk, :unc)
end
Test.DefaultTestSet("Unconstrained solvers", Any[Test.DefaultTestSet("Problem (x₁ - 1)² + 4(x₂ - 1)²", Any[], 4, false, false, true, 1.748902545639294e9, 1.748902548050721e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Problem Diagonal quadratic", Any[], 4, false, false, true, 1.748902548050747e9, 1.748902548990426e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Problem Non-zero residual", Any[], 4, false, false, true, 1.748902548990456e9, 1.748902549612826e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Problem Rosenbrock", Any[], 4, false, false, true, 1.748902549612868e9, 1.748902550103593e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Problem Overdetermined", Any[], 4, false, false, true, 1.74890255010362e9, 1.748902550685405e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Problem Extended Rosenbrock", Any[], 4, false, false, true, 1.748902550685433e9, 1.748902551668328e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/unconstrained.jl"), Test.DefaultTestSet("Precision Float16 for ptype unc", Any[], 7, false, false, true, 1.748902552196215e9, 1.748902555741578e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype unc", Any[], 7, false, false, true, 1.748902555741625e9, 1.748902558964976e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype unc", Any[], 7, false, false, true, 1.748902558965023e9, 1.74890255974633e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype unc", Any[], 7, false, false, true, 1.748902559746368e9, 1.748902563511495e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl")], 0, false, false, true, 1.748902539438955e9, 1.74890256351152e9, false, "none")

Bound-constrained Optimization

The function bound_constrained_nlp_set returns a list of optimization problems in the ADNLPModel format.

Secondly, the function bound_constrained_nlp will test the behavior of the JSO-compliant function solver on the test set. More precisely, it will test that primal and dual feasibility at the solution (stats.solution) computed with kkt_checker and that stats.primal_feas and stats.dual_feas are relatively small, i.e. smaller than atol + rtol * ‖∇f(x⁰)‖ where atol = rtol = 1e-6. It will also be tested that the final status stats.status is :first_order. The problems are sufficiently simple that we expect the solver to solve them successfully.

using SolverTest
using Test
using JSOSolvers # export solver trunk
@testset "Bound-constrained solvers" begin
  bound_constrained_nlp(tron)
  multiprecision_nlp(tron, :unc)
  multiprecision_nlp(tron, :bnd)
end
Test.DefaultTestSet("Bound-constrained solvers", Any[Test.DefaultTestSet("Problem Simple quadratic", Any[], 4, false, false, true, 1.748902566680724e9, 1.748902568777074e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Problem Rosenbrock inactive bounds", Any[], 4, false, false, true, 1.748902568777106e9, 1.748902569414618e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Problem Rosenbrock active bounds", Any[], 4, false, false, true, 1.748902569414649e9, 1.748902570017555e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Problem One fixed variable", Any[], 4, false, false, true, 1.748902570017584e9, 1.748902570693584e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Problem All variables fixed", Any[], 4, false, false, true, 1.74890257069361e9, 1.748902571106911e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Problem Extended Rosenbrock", Any[], 4, false, false, true, 1.748902571106942e9, 1.748902571925207e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/bound-constrained.jl"), Test.DefaultTestSet("Precision Float16 for ptype unc", Any[], 7, false, false, true, 1.748902572453092e9, 1.748902574936371e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype unc", Any[], 7, false, false, true, 1.748902574936413e9, 1.74890257693933e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype unc", Any[], 7, false, false, true, 1.748902576939371e9, 1.748902577251734e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype unc", Any[], 7, false, false, true, 1.748902577251769e9, 1.748902579473099e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float16 for ptype bnd", Any[], 7, false, false, true, 1.74890257947315e9, 1.74890257951169e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype bnd", Any[], 7, false, false, true, 1.748902579511717e9, 1.748902579549012e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype bnd", Any[], 7, false, false, true, 1.748902579549037e9, 1.748902579552427e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype bnd", Any[], 7, false, false, true, 1.748902579552449e9, 1.748902579590871e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl")], 0, false, false, true, 1.748902563517341e9, 1.748902579590891e9, false, "none")

Similarly, bound_constrained_nls_set and bound_constrained_nls are preparing the same tests for nonlinear least squares.

using SolverTest
using Test
using JSOSolvers # export solver trunk
@testset "Bound-constrained solvers" begin
  bound_constrained_nls(tron)
  multiprecision_nls(tron, :unc)
  multiprecision_nls(tron, :bnd)
end
Test.DefaultTestSet("Bound-constrained solvers", Any[Test.DefaultTestSet("Problem Simple quadratic", Any[], 4, false, false, true, 1.748902584703623e9, 1.748902586499086e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem Non-zero residual", Any[], 4, false, false, true, 1.748902586499115e9, 1.748902586993501e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem Rosenbrock inactive bounds", Any[], 4, false, false, true, 1.748902586993527e9, 1.748902587539082e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem Rosenbrock active bounds", Any[], 4, false, false, true, 1.74890258753911e9, 1.748902588040311e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem One fixed variable", Any[], 4, false, false, true, 1.748902588040338e9, 1.748902588550018e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem All variables fixed", Any[], 4, false, false, true, 1.748902588550043e9, 1.748902589240384e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Problem Extended Rosenbrock", Any[], 4, false, false, true, 1.748902589240418e9, 1.748902590044278e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/bound-constrained.jl"), Test.DefaultTestSet("Precision Float16 for ptype unc", Any[], 7, false, false, true, 1.748902590407625e9, 1.748902592295967e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype unc", Any[], 7, false, false, true, 1.748902592296009e9, 1.74890259391064e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype unc", Any[], 7, false, false, true, 1.74890259391068e9, 1.74890259405393e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype unc", Any[], 7, false, false, true, 1.748902594053962e9, 1.748902595949794e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float16 for ptype bnd", Any[], 7, false, false, true, 1.748902595949862e9, 1.748902596661919e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype bnd", Any[], 7, false, false, true, 1.748902596661954e9, 1.748902597443073e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype bnd", Any[], 7, false, false, true, 1.748902597443111e9, 1.748902598175777e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl"), Test.DefaultTestSet("Precision BigFloat for ptype bnd", Any[], 7, false, false, true, 1.748902598175813e9, 1.748902599207933e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nls/multiprecision.jl")], 0, false, false, true, 1.748902579596568e9, 1.748902599207954e9, false, "none")

Equality-constrained Optimization

This expands straightforwardly to equality-constrained problems.

using SolverTest
using Test
using DCISolver # export solver dci
@testset "Testing DCI-solver" begin

  @testset "$foo" for foo in [
    unconstrained_nlp,
    equality_constrained_nlp,
  ]
    foo(dci)
  end

  @testset "Multiprecision tests" begin
    for ptype in [:unc, :equ]
      multiprecision_nlp(
        dci,
        ptype,
        precisions = (Float32, Float64),
      )
    end
  end
end
Test.DefaultTestSet("Testing DCI-solver", Any[Test.DefaultTestSet("unconstrained_nlp", Any[Test.DefaultTestSet("Problem (x₁ - 1)² + 4(x₂ - 1)²", Any[], 4, false, false, true, 1.748902601617492e9, 1.748902606663306e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Diagonal quadratic", Any[], 4, false, false, true, 1.748902606663328e9, 1.748902608410626e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Tridiagonal quadratic", Any[], 4, false, false, true, 1.74890260841065e9, 1.748902608980254e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Rosenbrock", Any[], 4, false, false, true, 1.748902608980277e9, 1.748902608997387e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl"), Test.DefaultTestSet("Problem Extended Rosenbrock", Any[], 4, false, false, true, 1.748902608997407e9, 1.748902609231403e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/unconstrained.jl")], 0, false, false, true, 1.748902600655171e9, 1.748902609231408e9, false, "none"), Test.DefaultTestSet("equality_constrained_nlp", Any[Test.DefaultTestSet("Problem Simple quadratic problem", Any[], 5, false, false, true, 1.748902611620336e9, 1.74890261516934e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/equality-constrained.jl"), Test.DefaultTestSet("Problem HS6", Any[], 5, false, false, true, 1.748902615169372e9, 1.748902615972266e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/equality-constrained.jl"), Test.DefaultTestSet("Problem scaled HS8", Any[], 5, false, false, true, 1.748902615972288e9, 1.748902616760093e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/equality-constrained.jl"), Test.DefaultTestSet("Problem ‖x‖² s.t. ∑x = n", Any[], 5, false, false, true, 1.748902616760116e9, 1.748902618168961e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/equality-constrained.jl"), Test.DefaultTestSet("Problem Rosenbrock with ∑x = 2", Any[], 5, false, false, true, 1.748902618168993e9, 1.748902619125583e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/equality-constrained.jl")], 0, false, false, true, 1.748902609231445e9, 1.748902619125587e9, false, "none"), Test.DefaultTestSet("Multiprecision tests", Any[Test.DefaultTestSet("Precision Float32 for ptype unc", Any[], 7, false, false, true, 1.748902619125636e9, 1.7489026244894e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype unc", Any[], 7, false, false, true, 1.748902624489434e9, 1.748902625241644e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float32 for ptype equ", Any[], 7, false, false, true, 1.748902625241678e9, 1.748902629284789e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl"), Test.DefaultTestSet("Precision Float64 for ptype equ", Any[], 7, false, false, true, 1.748902629284828e9, 1.748902629967965e9, false, "/home/runner/work/SolverTest.jl/SolverTest.jl/src/nlp/multiprecision.jl")], 0, false, false, true, 1.74890261912561e9, 1.748902629967969e9, false, "none")], 0, false, false, true, 1.748902600655075e9, 1.74890262996797e9, false, "none")

Bound and Equality-constrained Optimization

This is still work in progress, see Issue 8.

Inequality-constrained optimization

This is still work in progress, see Issue 8.