Pick the best models for your questions, for free¶
You have a task, a few candidate models and a few ways to prompt them. Which combination answers best, and which answers fast enough? This guide runs every combination on your own questions, scores the answers, and tells you.
It uses only free model endpoints, so it costs nothing but time.
1. Install¶
pip install "frontierkit[real]"
git clone https://github.com/frontierkit/frontierkit
cd frontierkit
The real extra adds the OpenAI SDK, which is used to talk to every vendor
below; none of them is OpenAI.
2. Get free keys¶
Any one is enough; more vendors means more to compare.
| Vendor | Where | Variable |
|---|---|---|
| SAIA (GWDG Chat AI) | Academic Cloud / KISSKI | SAIA_API_KEY |
| NVIDIA API catalog | build.nvidia.com | NVIDIA_API_KEY |
| Groq | console.groq.com | GROQ_API_KEY |
| OpenRouter | openrouter.ai | OPENROUTER_API_KEY |
Set them as environment variables. On Windows, search the Start menu for "Edit environment variables for your account" and add them under User variables, then open a new terminal. On macOS or Linux:
export SAIA_API_KEY=...
Vendors rename and retire models often. If a default model has gone, the
script skips it and prints the model names your key can use; set the matching
variable, such as GROQ_MODEL, to one of them.
3. Write your questions¶
Make a CSV file with a question column, an answer column, and optionally a
context column. Separate acceptable answers with |:
question,answer,context
Which organ produces insulin?,pancreas,"Insulin is a hormone made by beta cells in the pancreas. ..."
What is the largest ocean on Earth?,Pacific Ocean|Pacific,"The Pacific Ocean is the largest ..."
- Use questions like the ones your application will really get. Ten to thirty is a good start.
- Keep reference answers short. The default score compares words, so a one-phrase reference judges a one-phrase answer fairly.
- Add a
contextcolumn if your application gives the model reading material. The script then compares answering with and without it.
examples/pick_models_questions.csv has eight general-knowledge questions to
try first.
4. Run it¶
python examples/pick_models.py --questions my_questions.csv
It first checks each model with one call and skips any that fail, printing the vendor's error and, where it can, the model names your key may use. Then it tries every combination and prints progress as it goes.
The number of calls is models × prompt styles × context options × questions.
With three models, two prompt styles, context on and off, and 10 questions,
that is 120 calls. Calls are spaced to stay under free-tier limits, and some
free endpoints take seconds or even minutes per answer, so start with
--max-questions 5.
5. Read the results¶
Each row is one setup:
| Column | Meaning |
|---|---|
| quality | mean score over your questions, from 0 to 1 |
| likely range | where the true quality plausibly lies, given how few questions were asked |
| answered | share of questions that got any answer |
| latency | mean time per answer |
* |
not beaten on both quality and speed by any other setup |
Below the table the script says one of two things:
- Best quality, when one setup is ahead of every other by more than chance could explain.
- Too close to call, when your questions cannot tell the top setup apart from others. It lists those setups and the gap to each with its likely range, and names a faster one if it scored within 0.05 of the top. The questions do not settle the order of these setups; add questions before relying on it.
If the top score is flattered by having been picked from many setups, the script also says what to expect on new questions.
The ranges are honest about small samples, so with a handful of questions they are wide: at 12 questions a range spans about 0.5. Around 30 questions separate large gaps (0.4 and more); gaps of 0.1 need a few hundred. A failed call counts as 0, which lowers a setup's quality and widens its range, so run a setup again if it shows failures.
The default score is word overlap with your reference (--scorer f1). Use
--scorer contains when answers may be long but must mention the reference,
or --scorer exact for labels and numbers.
6. Compare your own options¶
Everything compared is listed near the top of examples/pick_models.py:
PROMPT_STYLES: prompt templates, with{question}where the question goes.CONTEXT_OPTIONS: whether the context column is included.MODELS: one row per model, with its vendor's base URL and the gap kept between calls. Any vendor or local server that accepts OpenAI-style chat requests works, Ollama included.
Your own code can be a step too. Each step is a plain Python function that receives the previous step's output: a retriever over your documents, a re-ranker, a check on the answer. See "Run your own functions" in the README.
7. When there are too many combinations¶
Trying everything is the right choice while it stays affordable. When it
doesn't, let FrontierKit choose what to try. run_closed_loop with an eval set
scores a few setups per round, estimates how good each component is from the
scores, and proposes the next setups from those estimates; explore=N makes it
also try untried setups each round.
On FrontierKit's own question-answering benchmark this finds the best setup in
about the same number of calls as picking setups at random, so it is not yet a
reason on its own to prefer the loop over trying everything; see
benchmarks/qa_frontier/.
An example result¶
On 12 questions from the SQuAD reading-comprehension set, with three free models
and three amounts of context, the benchmark in benchmarks/qa_frontier/
measured (word-overlap score, mean latency per answer):
| Model | Context | Quality | Latency |
|---|---|---|---|
| Gemma 31B on SAIA | best paragraph | 0.95 | 230 ms |
| Gemma 31B on SAIA | best three paragraphs | 0.95 | 266 ms |
| Gemma 31B on NVIDIA | best paragraph | 0.95 | 14,516 ms |
| Llama 8B on SAIA | best paragraph | 0.33 | 250 ms |
| Gemma 31B on SAIA | none | 0.10 | 259 ms |
| Llama 8B on SAIA | none | 0.00 | 247 ms |
Three things stand out: without reading material even the strong model gets almost nothing right; the small model is no faster here and much worse; and the same model was over 60 times faster on one vendor than on the other. Your questions and vendors will give different numbers, which is the reason to measure.