Skip to content

Benchmarks

Three sets of measurements ship with the project. The first two run on free models and measure answers. The third measures the search strategies alone.

Which setup answers best

benchmarks/qa_frontier compares pipelines of three steps on reading-comprehension questions from SQuAD: how much reading material the model gets (none, the best-matching paragraph, or the best three), which free model answers, and whether the reply is kept whole or cut to its first clause. Answers are scored with the SQuAD token F1 against the reference answers.

Thirty questions and two free models on SAIA, with the reply kept whole:

Setup Quality Likely range Time per answer Against the top row
Gemma 4 31B, three passages 0.95 0.74 to 1.00 362 ms
Gemma 4 31B, one passage 0.90 0.69 to 1.00 285 ms too close to call
Llama 3.1 8B, three passages 0.34 0.12 to 0.48 247 ms worse
Llama 3.1 8B, one passage 0.30 0.11 to 0.50 259 ms worse
Gemma 4 31B, no passages 0.14 0.01 to 0.35 257 ms worse
Llama 3.1 8B, no passages 0.01 0.00 to 0.22 225 ms worse

The first two setups cannot be told apart on thirty questions, and both clearly beat the rest. The ranges and verdicts come from mean_interval and paired_difference; see How sure is the result.

In an earlier run on twelve SQuAD questions, the same Gemma model scored 0.95 on two vendors but answered in 230 ms on one and 14,516 ms on the other. Without reading material it scored 0.10, and a smaller model that was no faster scored 0.33.

Run it yourself with any of the free keys. Every reply is kept, so collection can stop and resume around rate limits, and scoring runs offline:

python -m benchmarks.qa_frontier.collect --budget 100
python -m benchmarks.qa_frontier.report
python -m benchmarks.qa_frontier.replay

The last command replays the kept replies to compare ways of finding the best pipelines within a budget of calls: trying every pipeline in order, trying them at random, and run_closed_loop. The closed loop currently finds the best pipeline in about as many calls as random picking.

Does the best setup stay the best

benchmarks/drift asks the same thirty questions of the same free models every day and reports whether the setup chosen on the first day still holds. Hosted models change without notice, and this measures how often that changes the choice.

python -m benchmarks.drift.collect
python -m benchmarks.drift.report

For each day the report prints the best setup, its quality as measured and corrected for having been picked from many, how many setups are too close to it to call, and the failed calls per model. It needs a sample a day over several weeks, so no results are shown here.

Search on declared numbers

The rest of this page is about a different question: how well the search strategies recover the best pipelines from declared cost, latency and reliability alone, before anything is run. Nothing here measures answers.

Setup

The search space is a five-step pipeline (retrieval, generation, verification, refinement, postprocessing) with four components per step, so 4**5 = 1,024 pipelines. Its exact frontier holds 27 of them.

Strategies compared: HeuristicSynthesizer (greedy), RandomSynthesizer, BeamSynthesizer (a beam on one weighted score), ParetoPartialBeamSynthesizer and ProgressiveParetoBeamSynthesizer.

Each strategy may evaluate a limited number of candidates: 20, 40, 80, 120 or 180. Beam searches were also run at widths 3, 5, 7, 10 and 15.

Two figures are reported:

  • Coverage: the share of the exact frontier the strategy recovered.
  • Hypervolume loss: how much of the space covered by the exact frontier the recovered one misses. Zero is a full recovery.

Measured tables

The tables below are generated, not hand-written. They were measured by scripts/regenerate_benchmarks.py on version 0.6.0 (Python 3.10.4). Random baselines use seeds 42 to 46; hypervolume estimates use 200,000 Monte Carlo samples with seed 42. Re-run them with:

python scripts/regenerate_benchmarks.py

Coverage by evaluation budget (beam width 7, 5-stage demo registry)

Coverage of the exact Pareto frontier under a bounded evaluation budget; Random cells are mean +/- std over 5 seeds (42-46).

Algorithm 20 40 80 120 180
Greedy 0.04 0.04 0.04 0.04 0.04
Random 0.04 +/- 0.03 0.11 +/- 0.04 0.20 +/- 0.06 0.31 +/- 0.03 0.39 +/- 0.11
ScalarBeam 0.15 0.19 0.19 0.19 0.19
ParetoPartialBeam 0.33 0.33 0.33 0.33 0.33
ProgressiveParetoBeam 0.74 1.00 1.00 1.00 1.00

Mean hypervolume loss by evaluation budget (beam width 7)

Monte Carlo hypervolume loss against the exact frontier (200000 samples, seed 42); Random cells are means over 5 seeds.

Algorithm 20 40 80 120 180
Greedy 115.5990 115.5990 115.5990 115.5990 115.5990
Random 43.2480 28.4835 20.1942 18.9936 16.7583
ScalarBeam 75.7605 68.4000 68.4000 68.4000 68.4000
ParetoPartialBeam 6.9060 6.9060 6.9060 6.9060 6.9060
ProgressiveParetoBeam 38.2215 0.0000 0.0000 0.0000 0.0000

Coverage by beam width (evaluation budget 120)

Coverage of the exact Pareto frontier against a fixed evaluation budget.

Algorithm 3 5 7 10 15
ScalarBeam 0.19 0.19 0.19 0.26 0.26
ParetoPartialBeam 0.22 0.26 0.33 0.63 0.85
ProgressiveParetoBeam 0.85 1.00 1.00 1.00 1.00

Mean hypervolume loss by beam width (evaluation budget 120)

Monte Carlo hypervolume loss against the exact frontier (200000 samples, seed 42).

Algorithm 3 5 7 10 15
ScalarBeam 83.9265 83.9265 68.4000 68.4000 68.4000
ParetoPartialBeam 23.2560 13.2660 6.9060 0.1005 0.0135
ProgressiveParetoBeam 1.9695 0.0000 0.0000 0.0000 0.0000

What the tables show

  • A beam on one weighted score prunes too early. It drops partial pipelines that would have ended on the frontier, and no later step can bring them back.
  • Keeping the partial pipelines nothing beats at each step recovers more, and always keeping the cheapest, fastest and most reliable ones protects the corners of the frontier.
  • Widening the beam as the pipeline grows recovers the whole frontier from a budget of 40 candidates at width 7, and from width 5 at a budget of 120. The plain beam stays at 19% in both cases.
  • Random search improves slowly with budget and reaches 39% at 180 candidates.

These results concern declared numbers. Which pipeline gives the best answers has to be measured, which is what the first two benchmarks do.