Tutorial

In this tutorial we illustrate the main uses of SolverBenchmark.

First, let's create fake data. It is imperative that the data for each solver be stored in DataFrames, and the collection of different solver must be stored in a dictionary of Symbol to DataFrame.

In our examples we'll use the following data.

using DataFrames, Printf, Random

Random.seed!(0)

n = 10
names = [:alpha, :beta, :gamma]
stats = Dict(name => DataFrame(:id => 1:n,
         :name => [@sprintf("prob%03d", i) for i = 1:n],
         :status => map(x -> x < 0.75 ? :first_order : :failure, rand(n)),
         :f => randn(n),
         :t => 1e-3 .+ rand(n) * 1000,
         :iter => rand(10:10:100, n),
         :irrelevant => randn(n)) for name in names)

Dict{Symbol,DataFrames.DataFrame} with 3 entries:
  :alpha => 10×7 DataFrame. Omitted printing of 1 columns…
  :beta  => 10×7 DataFrame. Omitted printing of 1 columns…
  :gamma => 10×7 DataFrame. Omitted printing of 1 columns…

The data consists of a (fake) run of three solvers alpha, beta and gamma. Each solver has a column id, which is necessary for joining the solvers (names can be repeated), and columns name, status, f, t and iter corresponding to problem results. There is also a column irrelevant with extra information that will not be used to produce our benchmarks.

Here are the statistics of solver alpha:

stats[:alpha]

10 rows × 7 columns

Tables

The first thing we may want to do is produce a table for each solver. Notice that the solver result is already a DataFrame, so there are a few options available in other packages, as well as simply printing the DataFrame. Our concern here is two-fold: producing publication-ready LaTeX tables, and web-ready markdown tables.

The simplest use is pretty_stats(io, dataframe). By default, io is stdout:

using SolverBenchmark

pretty_stats(stats[:alpha])

┌────────┬─────────┬─────────────┬───────────┬───────────┬────────┬────────────┐
│     id │    name │      status │         f │         t │   iter │ irrelevant │
├────────┼─────────┼─────────────┼───────────┼───────────┼────────┼────────────┤
│      1 │ prob001 │     failure │ -6.89e-01 │  6.24e+01 │     70 │   8.91e-01 │
│      2 │ prob002 │     failure │ -7.63e-01 │  3.53e+02 │     10 │  -2.29e+00 │
│      3 │ prob003 │ first_order │  3.97e-01 │  7.68e+02 │     10 │  -1.17e+00 │
│      4 │ prob004 │ first_order │  8.12e-01 │  4.31e+01 │     80 │  -8.80e-01 │
│      5 │ prob005 │ first_order │ -3.46e-01 │  2.68e+02 │     30 │  -4.94e-01 │
│      6 │ prob006 │ first_order │ -1.88e-01 │  6.68e+01 │     80 │   5.12e-01 │
│      7 │ prob007 │ first_order │ -1.61e+00 │  1.57e+02 │     60 │  -7.59e-01 │
│      8 │ prob008 │ first_order │ -2.48e+00 │  6.05e+02 │     40 │  -7.68e-01 │
│      9 │ prob009 │ first_order │  2.28e+00 │  1.36e+02 │     40 │  -3.03e-02 │
│     10 │ prob010 │     failure │  2.20e-01 │  8.38e+02 │     50 │  -1.32e+00 │
└────────┴─────────┴─────────────┴───────────┴───────────┴────────┴────────────┘

Printing is LaTeX format is achieved with pretty_latex_stats:

pretty_latex_stats(stats[:alpha])

\begin{longtable}{rrrrrrr}
\hline
id & name & status & f & t & iter & irrelevant \\\hline
\endhead
\hline
\multicolumn{7}{r}{{\bfseries Continued on next page}}\\
\hline
\endfoot
\endlastfoot
\(     1\) & prob001 & failure & \(-6.89\)e\(-01\) & \( 6.24\)e\(+01\) & \(    70\) & \( 8.91\)e\(-01\) \\
\(     2\) & prob002 & failure & \(-7.63\)e\(-01\) & \( 3.53\)e\(+02\) & \(    10\) & \(-2.29\)e\(+00\) \\
\(     3\) & prob003 & first\_order & \( 3.97\)e\(-01\) & \( 7.68\)e\(+02\) & \(    10\) & \(-1.17\)e\(+00\) \\
\(     4\) & prob004 & first\_order & \( 8.12\)e\(-01\) & \( 4.31\)e\(+01\) & \(    80\) & \(-8.80\)e\(-01\) \\
\(     5\) & prob005 & first\_order & \(-3.46\)e\(-01\) & \( 2.68\)e\(+02\) & \(    30\) & \(-4.94\)e\(-01\) \\
\(     6\) & prob006 & first\_order & \(-1.88\)e\(-01\) & \( 6.68\)e\(+01\) & \(    80\) & \( 5.12\)e\(-01\) \\
\(     7\) & prob007 & first\_order & \(-1.61\)e\(+00\) & \( 1.57\)e\(+02\) & \(    60\) & \(-7.59\)e\(-01\) \\
\(     8\) & prob008 & first\_order & \(-2.48\)e\(+00\) & \( 6.05\)e\(+02\) & \(    40\) & \(-7.68\)e\(-01\) \\
\(     9\) & prob009 & first\_order & \( 2.28\)e\(+00\) & \( 1.36\)e\(+02\) & \(    40\) & \(-3.03\)e\(-02\) \\
\(    10\) & prob010 & failure & \( 2.20\)e\(-01\) & \( 8.38\)e\(+02\) & \(    50\) & \(-1.32\)e\(+00\) \\\hline
\end{longtable}

Alternatively, you can print to a file.

open("alpha.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io, stats[:alpha])
  println(io, "\\end{document}")
end

run(`latexmk -quiet -pdf alpha.tex`)
run(`pdf2svg alpha.pdf alpha.svg`)

Process(`pdf2svg alpha.pdf alpha.svg`, ProcessExited(0))

If only a subset of columns should be printed, the DataFrame should be indexed accordingly:

df = stats[:alpha]
pretty_stats(df[!, [:name, :f, :t]])

┌─────────┬───────────┬───────────┐
│    name │         f │         t │
├─────────┼───────────┼───────────┤
│ prob001 │ -6.89e-01 │  6.24e+01 │
│ prob002 │ -7.63e-01 │  3.53e+02 │
│ prob003 │  3.97e-01 │  7.68e+02 │
│ prob004 │  8.12e-01 │  4.31e+01 │
│ prob005 │ -3.46e-01 │  2.68e+02 │
│ prob006 │ -1.88e-01 │  6.68e+01 │
│ prob007 │ -1.61e+00 │  1.57e+02 │
│ prob008 │ -2.48e+00 │  6.05e+02 │
│ prob009 │  2.28e+00 │  1.36e+02 │
│ prob010 │  2.20e-01 │  8.38e+02 │
└─────────┴───────────┴───────────┘

Markdown tables may be generated by supplying the PrettyTables tf keyword argument to specify the table format:

pretty_stats(df[!, [:name, :f, :t]], tf=markdown)

|    name |         f |         t |
|---------|-----------|-----------|
| prob001 | -6.89e-01 |  6.24e+01 |
| prob002 | -7.63e-01 |  3.53e+02 |
| prob003 |  3.97e-01 |  7.68e+02 |
| prob004 |  8.12e-01 |  4.31e+01 |
| prob005 | -3.46e-01 |  2.68e+02 |
| prob006 | -1.88e-01 |  6.68e+01 |
| prob007 | -1.61e+00 |  1.57e+02 |
| prob008 | -2.48e+00 |  6.05e+02 |
| prob009 |  2.28e+00 |  1.36e+02 |
| prob010 |  2.20e-01 |  8.38e+02 |

All values of tf accepted by PrettyTables may be used in SolverBenchmark.

The fmt_override option overrides the formatting of a specific column. The argument should be a dictionary of Symbol to format strings, where the format string will be applied to each element of the column.

The hdr_override changes the column headers.

fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(stdout,
             df[!, [:name, :f, :t]],
             col_formatters = fmt_override,
             hdr_override = hdr_override)

┌─────────┬────────────┬──────────┐
│    Name │       f(x) │     Time │
├─────────┼────────────┼──────────┤
│ prob001 │ -6.889e-01 │ 00062.37 │
│ prob002 │ -7.628e-01 │ 00353.13 │
│ prob003 │ +3.975e-01 │ 00767.60 │
│ prob004 │ +8.116e-01 │ 00043.14 │
│ prob005 │ -3.464e-01 │ 00267.99 │
│ prob006 │ -1.876e-01 │ 00066.85 │
│ prob007 │ -1.607e+00 │ 00156.64 │
│ prob008 │ -2.481e+00 │ 00605.30 │
│ prob009 │ +2.276e+00 │ 00135.75 │
│ prob010 │ +2.197e-01 │ 00838.12 │
└─────────┴────────────┴──────────┘

While col_formatters is for simple format strings, the PrettyTables API lets us define more elaborate formatters in the form of functions:

fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
pretty_stats(df[!, [:name, :f, :t]],
             col_formatters = fmt_override,
             hdr_override = hdr_override,
             formatters = (v, i, j) -> begin
               if j == 3  # t is the 3rd column
                 vi = floor(Int, v)
                 minutes = div(vi, 60)
                 seconds = vi % 60
                 micros = round(Int, 1e6 * (v - vi))
                 @sprintf("%2dm %02ds %06dμs", minutes, seconds, micros)
               else
                 v
               end
             end)

┌─────────┬────────────┬──────────────────┐
│    Name │       f(x) │             Time │
├─────────┼────────────┼──────────────────┤
│ prob001 │ -6.889e-01 │  1m 02s 368558μs │
│ prob002 │ -7.628e-01 │  5m 53s 129773μs │
│ prob003 │ +3.975e-01 │ 12m 47s 602896μs │
│ prob004 │ +8.116e-01 │  0m 43s 142023μs │
│ prob005 │ -3.464e-01 │  4m 27s 985634μs │
│ prob006 │ -1.876e-01 │  1m 06s 847444μs │
│ prob007 │ -1.607e+00 │  2m 36s 637637μs │
│ prob008 │ -2.481e+00 │ 10m 05s 297740μs │
│ prob009 │ +2.276e+00 │  2m 15s 745559μs │
│ prob010 │ +2.197e-01 │ 13m 58s 118754μs │
└─────────┴────────────┴──────────────────┘

See the PrettyTables.jl documentation for more information.

When using LaTeX format, the output must be understood by LaTeX. By default, numerical data in the table is wrapped in inline math environments. But those math environments would interfere with our formatting of the time. Thus we must first disable them for the time column using col_formatters, and then apply the PrettyTables formatter as above:

fmt_override = Dict(:f => "%+10.3e",
                    :t => "%08.2f")
hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
open("alpha2.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io,
                    df[!, [:name, :status, :f, :t, :iter]],
                    col_formatters = Dict(:t => "%f"),  # disable default formatting of t
                    formatters = (v,i,j) -> begin
                      if j == 4
                        xi = floor(Int, v)
                        minutes = div(xi, 60)
                        seconds = xi % 60
                        micros = round(Int, 1e6 * (v - xi))
                        @sprintf("\\(%2d\\)m \\(%02d\\)s \\(%06d \\mu\\)s", minutes, seconds, micros)
                      else
                        v
                      end
                  end)
  println(io, "\\end{document}")
end

Joining tables

In some occasions, instead of/in addition to showing individual results, we show a table with the result of multiple solvers.

df = join(stats, [:f, :t])
pretty_stats(stdout, df)

┌────────┬───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐
│     id │   f_alpha │   t_alpha │    f_beta │    t_beta │   f_gamma │   t_gamma │
├────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤
│      1 │ -6.89e-01 │  6.24e+01 │ -1.05e+00 │  1.77e+02 │  6.34e-02 │  3.26e+01 │
│      2 │ -7.63e-01 │  3.53e+02 │  8.16e-01 │  8.00e+01 │  1.19e-01 │  6.86e+02 │
│      3 │  3.97e-01 │  7.68e+02 │  1.53e-01 │  6.82e+02 │  2.71e+00 │  8.41e+02 │
│      4 │  8.12e-01 │  4.31e+01 │ -3.30e-01 │  9.29e+02 │ -6.91e-01 │  1.90e+02 │
│      5 │ -3.46e-01 │  2.68e+02 │  1.44e+00 │  9.73e+02 │ -5.51e-02 │  1.62e+02 │
│      6 │ -1.88e-01 │  6.68e+01 │ -4.43e-01 │  6.52e+02 │  4.23e-01 │  8.97e+02 │
│      7 │ -1.61e+00 │  1.57e+02 │  1.10e+00 │  5.97e+02 │ -1.43e+00 │  9.54e+01 │
│      8 │ -2.48e+00 │  6.05e+02 │ -2.51e-01 │  4.79e+02 │ -4.50e-01 │  7.77e+02 │
│      9 │  2.28e+00 │  1.36e+02 │  2.92e-01 │  6.32e+01 │ -8.81e-01 │  8.68e+02 │
│     10 │  2.20e-01 │  8.38e+02 │ -3.47e+00 │  4.71e+02 │  1.08e+00 │  8.38e+02 │
└────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘

The column :id is used as guide on where to join. In addition, we may have repeated columns between the solvers. We convery that information with argument invariant_cols.

df = join(stats, [:f, :t], invariant_cols=[:name])
pretty_stats(stdout, df)

┌────────┬─────────┬───────────┬───────────┬───────────┬───────────┬───────────┬───────────┐
│     id │    name │   f_alpha │   t_alpha │    f_beta │    t_beta │   f_gamma │   t_gamma │
├────────┼─────────┼───────────┼───────────┼───────────┼───────────┼───────────┼───────────┤
│      1 │ prob001 │ -6.89e-01 │  6.24e+01 │ -1.05e+00 │  1.77e+02 │  6.34e-02 │  3.26e+01 │
│      2 │ prob002 │ -7.63e-01 │  3.53e+02 │  8.16e-01 │  8.00e+01 │  1.19e-01 │  6.86e+02 │
│      3 │ prob003 │  3.97e-01 │  7.68e+02 │  1.53e-01 │  6.82e+02 │  2.71e+00 │  8.41e+02 │
│      4 │ prob004 │  8.12e-01 │  4.31e+01 │ -3.30e-01 │  9.29e+02 │ -6.91e-01 │  1.90e+02 │
│      5 │ prob005 │ -3.46e-01 │  2.68e+02 │  1.44e+00 │  9.73e+02 │ -5.51e-02 │  1.62e+02 │
│      6 │ prob006 │ -1.88e-01 │  6.68e+01 │ -4.43e-01 │  6.52e+02 │  4.23e-01 │  8.97e+02 │
│      7 │ prob007 │ -1.61e+00 │  1.57e+02 │  1.10e+00 │  5.97e+02 │ -1.43e+00 │  9.54e+01 │
│      8 │ prob008 │ -2.48e+00 │  6.05e+02 │ -2.51e-01 │  4.79e+02 │ -4.50e-01 │  7.77e+02 │
│      9 │ prob009 │  2.28e+00 │  1.36e+02 │  2.92e-01 │  6.32e+01 │ -8.81e-01 │  8.68e+02 │
│     10 │ prob010 │  2.20e-01 │  8.38e+02 │ -3.47e+00 │  4.71e+02 │  1.08e+00 │  8.38e+02 │
└────────┴─────────┴───────────┴───────────┴───────────┴───────────┴───────────┴───────────┘

join also accepts hdr_override for changing the column name before appending _solver.

hdr_override = Dict(:name => "Name", :f => "f(x)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
pretty_stats(stdout, df)

┌────────┬─────────┬────────────┬────────────┬───────────┬───────────┬────────────┬────────────┐
│     id │    name │ f(x)_alpha │ Time_alpha │ f(x)_beta │ Time_beta │ f(x)_gamma │ Time_gamma │
├────────┼─────────┼────────────┼────────────┼───────────┼───────────┼────────────┼────────────┤
│      1 │ prob001 │  -6.89e-01 │   6.24e+01 │ -1.05e+00 │  1.77e+02 │   6.34e-02 │   3.26e+01 │
│      2 │ prob002 │  -7.63e-01 │   3.53e+02 │  8.16e-01 │  8.00e+01 │   1.19e-01 │   6.86e+02 │
│      3 │ prob003 │   3.97e-01 │   7.68e+02 │  1.53e-01 │  6.82e+02 │   2.71e+00 │   8.41e+02 │
│      4 │ prob004 │   8.12e-01 │   4.31e+01 │ -3.30e-01 │  9.29e+02 │  -6.91e-01 │   1.90e+02 │
│      5 │ prob005 │  -3.46e-01 │   2.68e+02 │  1.44e+00 │  9.73e+02 │  -5.51e-02 │   1.62e+02 │
│      6 │ prob006 │  -1.88e-01 │   6.68e+01 │ -4.43e-01 │  6.52e+02 │   4.23e-01 │   8.97e+02 │
│      7 │ prob007 │  -1.61e+00 │   1.57e+02 │  1.10e+00 │  5.97e+02 │  -1.43e+00 │   9.54e+01 │
│      8 │ prob008 │  -2.48e+00 │   6.05e+02 │ -2.51e-01 │  4.79e+02 │  -4.50e-01 │   7.77e+02 │
│      9 │ prob009 │   2.28e+00 │   1.36e+02 │  2.92e-01 │  6.32e+01 │  -8.81e-01 │   8.68e+02 │
│     10 │ prob010 │   2.20e-01 │   8.38e+02 │ -3.47e+00 │  4.71e+02 │   1.08e+00 │   8.38e+02 │
└────────┴─────────┴────────────┴────────────┴───────────┴───────────┴────────────┴────────────┘

hdr_override = Dict(:name => "Name", :f => "\\(f(x)\\)", :t => "Time")
df = join(stats, [:f, :t], invariant_cols=[:name], hdr_override=hdr_override)
open("alpha3.tex", "w") do io
  println(io, "\\documentclass[varwidth=20cm,crop=true]{standalone}")
  println(io, "\\usepackage{longtable}")
  println(io, "\\begin{document}")
  pretty_latex_stats(io, df)
  println(io, "\\end{document}")
end

Profiles

Performance profiles are a comparison tool developed by Dolan and Moré, 2002 that takes into account the relative performance of a solver and whether it has achieved convergence for each problem. SolverBenchmark.jl uses BenchmarkProfiles.jl for generating performance profiles from the dictionary of DataFrames.

The basic usage is performance_profile(stats, cost), where cost is a function applied to a DataFrame and returning a vector.

using Plots
pyplot()

p = performance_profile(stats, df -> df.t)
Plots.svg(p, "profile1")

Notice that we used df -> df.t which corresponds to the column :t of the DataFrames. This does not take into account that the solvers have failed for a few problems (according to column :status). The next profile takes that into account.

cost(df) = (df.status .!= :success) * Inf + df.t
p = performance_profile(stats, cost)
Plots.svg(p, "profile2")

Profile wall

Another profile function is profile_solvers, which creates a wall of performance profiles, accepting multiple costs and doing 1 vs 1 comparisons in addition to the traditional performance profile.

solved(df) = (df.status .== :success)
costs = [df -> .!solved(df) * Inf + df.t, df -> .!solved(df) * Inf + df.iter]
costnames = ["Time", "Iterations"]
p = profile_solvers(stats, costs, costnames)
Plots.svg(p, "profile3")