FrontierKit runs your questions through every model, prompt and pipeline step you want to compare. It tells you which setups are really better and which are too close to call, and lets you check weeks later that your choice still holds.
Each dot is a setup you compared. The line joins the ones nothing beats on quality, speed and cost. The bars show how far each score can be trusted.
The top two setups cannot be told apart on these questions, so choose between them on speed or cost. Both clearly beat the smaller model, and the same model without reading material.
Quality is word overlap with the reference answers, from 0 to 1. The bar is the range the true score plausibly lies in. Thirty reading-comprehension questions from SQuAD, answered by two free models.
A CSV of questions and reference answers is enough. Scorers cover short answers and extracted triples, or pass your own.
Every score comes with a range. Two setups are compared question by question, and the answer can be that they are too close to call.
You see the setups that are not beaten on all three, and pick the trade-off you want.
Every model reply is kept in a local file. Run the comparison again and no model is called; add one setup and only its new steps are.
Save what you chose and why, then verify it on a schedule. It reports whether quality dropped, the setup got slower, or the model is gone.
A rate limit, an outage or a retired model is recorded as what it is, not as a wrong answer.
Works with any service or local server that accepts OpenAI-style chat requests, including SAIA, NVIDIA, Groq, OpenRouter and Ollama. Each pipeline step can also be a plain Python function.
Twelve reading-comprehension questions from SQuAD, the same Gemma model on two vendors.
The same model with the same quality, 0.95, answered this fast on one vendor and this slowly on the other.
What happened to its quality when the reading material was taken away.
The score of a smaller model that was no faster.
See the benchmark behind these figures.
One package. A free key from SAIA, NVIDIA, Groq or OpenRouter is enough to start.
pip install "frontierkit[real]"
Run every combination of prompt, context and model on your own questions.
python examples/pick_models.py --questions my_questions.csv
Run the saved choice again later, by hand or on a schedule.
frontierkit verify decision.json --setup compare.py
FrontierKit is an alpha-stage research library. Its API may change between minor versions.
Free models, five minutes, and a result you can explain.