What happens now
Several fitting methods run more than one search at the same time, each from a different starting point, and keep the best result. This applies to gntr, lbfgs, trf, ms, profile_likelihood, powell, and sim. The number of starts comes from population_size for the gradient methods and from n_starts for powell and sim.
At the end of the run, PyBNF reports the single best parameter set it found. It does not report how the individual starts did. When a start finishes, the log records why it stopped but not the objective value it reached. See ConcurrentMultiStartOptimizer.got_result in pybnf/algorithms/optimizers/concurrent_multistart.py around line 258.
Why this is a problem
A user cannot tell whether the answer is trustworthy, because two very different runs look identical in the output.
- All twenty starts converged to roughly the same objective value. The fit has very likely found the best answer available, and running more starts would not help.
- All twenty starts landed somewhere different, and the reported answer is only the least bad of twenty poor results. More starts are needed, or the model or the parameter bounds need attention.
Today both cases print one number and nothing else, so a user has no basis for deciding whether to run the fit again with more starts.
Sorting the final objective values from every start and looking at the shape of the resulting curve is the usual way to answer this question in the parameter fitting literature, where it is often called a waterfall plot. A flat run of starts sitting at the same low value means the search found a consistent answer. A staircase with no flat section at the bottom means it did not.
What to add
At the end of a multi-start fit, write a small text file to the results directory, for example Results/multistart_summary.txt, with one row per start sorted by final objective value from best to worst. Suggested columns:
- rank
- start number
- final objective value
- reason the start stopped
- number of iterations it ran
Also print a short version to the screen at the end of the run, so a user sees it without opening a file.
Notes
The data already exists. Each start has a runner object holding its final objective value (runner.fval), its stop reason (runner.stop_reason), and its iteration count (runner.iteration). All of these are available at the point in got_result where a start finishes. This is a reporting change and does not change how any fitting method searches.
The metaheuristic methods de, ade, ss, and pso also support multiple starts through n_starts, but they run their starts one after another rather than at the same time. The same summary would be just as useful there. It could go in the same change or be left for a follow up.
What happens now
Several fitting methods run more than one search at the same time, each from a different starting point, and keep the best result. This applies to
gntr,lbfgs,trf,ms,profile_likelihood,powell, andsim. The number of starts comes frompopulation_sizefor the gradient methods and fromn_startsforpowellandsim.At the end of the run, PyBNF reports the single best parameter set it found. It does not report how the individual starts did. When a start finishes, the log records why it stopped but not the objective value it reached. See
ConcurrentMultiStartOptimizer.got_resultinpybnf/algorithms/optimizers/concurrent_multistart.pyaround line 258.Why this is a problem
A user cannot tell whether the answer is trustworthy, because two very different runs look identical in the output.
Today both cases print one number and nothing else, so a user has no basis for deciding whether to run the fit again with more starts.
Sorting the final objective values from every start and looking at the shape of the resulting curve is the usual way to answer this question in the parameter fitting literature, where it is often called a waterfall plot. A flat run of starts sitting at the same low value means the search found a consistent answer. A staircase with no flat section at the bottom means it did not.
What to add
At the end of a multi-start fit, write a small text file to the results directory, for example
Results/multistart_summary.txt, with one row per start sorted by final objective value from best to worst. Suggested columns:Also print a short version to the screen at the end of the run, so a user sees it without opening a file.
Notes
The data already exists. Each start has a runner object holding its final objective value (
runner.fval), its stop reason (runner.stop_reason), and its iteration count (runner.iteration). All of these are available at the point ingot_resultwhere a start finishes. This is a reporting change and does not change how any fitting method searches.The metaheuristic methods
de,ade,ss, andpsoalso support multiple starts throughn_starts, but they run their starts one after another rather than at the same time. The same summary would be just as useful there. It could go in the same change or be left for a follow up.