Open source, MIT licence, Python 3.10 and newer

Find the setup that answers your questions best. Know how sure you can be.

FrontierKit runs your questions through every model, prompt and pipeline step you want to compare. It tells you which setups are really better and which are too close to call, and lets you check weeks later that your choice still holds.

$pip install frontierkit
The frontier
not beatenbeatenlikely range
quality cost and time

Each dot is a setup you compared. The line joins the ones nothing beats on quality, speed and cost. The bars show how far each score can be trusted.

Thirty questions, two free models

A leaderboard names a winner. FrontierKit says what your questions can support.

The top two setups cannot be told apart on these questions, so choose between them on speed or cost. Both clearly beat the smaller model, and the same model without reading material.

Gemma 4 31B, three passages362 ms per answer
0.95 0.74 to 1.00
best measured
Gemma 4 31B, one passage285 ms per answer
0.90 0.69 to 1.00
too close to call
Llama 3.1 8B, three passages247 ms per answer
0.34 0.12 to 0.48
worse
Llama 3.1 8B, one passage259 ms per answer
0.30 0.11 to 0.50
worse
Gemma 4 31B, no passages257 ms per answer
0.14 0.01 to 0.35
worse
Llama 3.1 8B, no passages225 ms per answer
0.01 0.00 to 0.22
worse

Quality is word overlap with the reference answers, from 0 to 1. The bar is the range the true score plausibly lies in. Thirty reading-comprehension questions from SQuAD, answered by two free models.

What it does

Built for the choice you have to defend.

Your questions, not a leaderboard

A CSV of questions and reference answers is enough. Scorers cover short answers and extracted triples, or pass your own.

It says how sure it is

Every score comes with a range. Two setups are compared question by question, and the answer can be that they are too close to call.

Quality, speed and cost together

You see the setups that are not beaten on all three, and pick the trade-off you want.

No answer is paid for twice

Every model reply is kept in a local file. Run the comparison again and no model is called; add one setup and only its new steps are.

A choice you can check again

Save what you chose and why, then verify it on a schedule. It reports whether quality dropped, the setup got slower, or the model is gone.

Failures are named, not hidden

A rate limit, an outage or a retired model is recorded as what it is, not as a wrong answer.

Works with any service or local server that accepts OpenAI-style chat requests, including SAIA, NVIDIA, Groq, OpenRouter and Ollama. Each pipeline step can also be a plain Python function.

Why measure

The model card will not tell you this.

Twelve reading-comprehension questions from SQuAD, the same Gemma model on two vendors.

230 or 14,516 ms

The same model with the same quality, 0.95, answered this fast on one vendor and this slowly on the other.

0.95 to 0.10

What happened to its quality when the reading material was taken away.

0.33

The score of a smaller model that was no faster.

See the benchmark behind these figures.

Three steps

From your questions to a choice that holds.

1. Install

One package. A free key from SAIA, NVIDIA, Groq or OpenRouter is enough to start.

pip install "frontierkit[real]"
2. Compare

Run every combination of prompt, context and model on your own questions.

python examples/pick_models.py --questions my_questions.csv
3. Check it still holds

Run the saved choice again later, by hand or on a schedule.

frontierkit verify decision.json --setup compare.py
Where it stands

An honest status report.

FrontierKit is an alpha-stage research library. Its API may change between minor versions.

Try it on your own questions.

Free models, five minutes, and a result you can explain.