Using FrontierKit from Python¶
The get-started guide runs a ready-made script. This page shows the pieces that script is built from, so you can compare anything: your own retrievers, prompts, checkers and models.
Everything here is importable from frontierkit unless noted.
The three ideas¶
- A component is one option for one step of a pipeline: a model, a prompt
template, a retriever. It is described by a
ComponentMetadatarecord and kept in aCapabilityRegistry. - A task (
TaskSpecification) names the steps a pipeline needs, in order, plus optional limits on cost, latency and privacy. - A pipeline is one component per step. FrontierKit lists or searches the possible pipelines, runs them, and measures what they do.
from frontierkit import CapabilityRegistry, ComponentMetadata, TaskSpecification
registry = CapabilityRegistry()
registry.register(ComponentMetadata(
name="short_prompt", component_type="tool", capabilities=["prompt"],
cost_per_call=0.0, avg_latency_ms=0.0, reliability_score=1.0,
privacy_level="internal",
))
# ... one record per option ...
task = TaskSpecification(required_capabilities=["prompt", "answer"])
The declared numbers are starting guesses. They matter less once you measure.
Run your own functions¶
Back each component with a plain Python function. Each function receives the previous step's output and returns its own.
from frontierkit import AgentExecutor, CallableBackend
backend = CallableBackend({
"retrieve_keyword": lambda question: search(question),
"answer_short": lambda passages: draft(passages),
"verify_grounded": lambda answer: check(answer),
})
backend.missing(registry) # [] when every registry component is covered
result = AgentExecutor(backend=backend).execute(pipeline.graph, task_input="your question")
result.output # what the last step returned
result.success # False if any step raised or reported failure
Latency is measured around each call. A function that raises, or returns
StepResult(output=None, success=False), is recorded as a failed step and
stops the pipeline. Cost is the declared cost_per_call unless the function
returns StepResult(output, cost=...).
Runnable version: examples/callable_pipeline.py.
Call models from any vendor¶
OpenAIBackend talks to any vendor or local server that accepts OpenAI-style
chat requests. Install the extra first: pip install "frontierkit[real]".
import os
from frontierkit import RoutedBackend
from frontierkit.adapters.openai_backend import OpenAIBackend
backend = RoutedBackend({
"model_a": OpenAIBackend(model="model-a", base_url="https://vendor-a.example/v1",
api_key=os.environ["VENDOR_A_KEY"]),
"model_b": OpenAIBackend(model="model-b", base_url="https://vendor-b.example/v1",
api_key=os.environ["VENDOR_B_KEY"]),
}, default=CallableBackend({...})) # local steps go to the default
RoutedBackendsends each component to its own backend, so one comparison can mix vendors and local functions.request_options={"max_tokens": 300, "temperature": 0}is sent with every request. Vendors differ in what they accept: current Claude models, for example, reject atemperaturebelow 1.- Read keys from the environment. Do not write them into source files.
Measure cost from token usage¶
Give the backend your prices and each call is costed from the token counts the vendor reports:
from frontierkit.adapters.openai_backend import OpenAIBackend, TokenPrice
backend = OpenAIBackend(
model="your-model",
prices={"your-model": TokenPrice(input_per_million=2.0, output_per_million=8.0)},
)
Prices are yours to supply because they change. A model without a price
records the declared cost_per_call.
Score pipelines on your examples¶
An EvalSet pairs example inputs with a scorer. evaluate_pipelines runs each
pipeline on every example and reports what was measured.
from frontierkit import (
EvalSet, Example, enumerate_pipelines, evaluate_pipelines,
quality_frontier, token_f1,
)
eval_set = EvalSet(
examples=[
Example("Which organ produces insulin?", expected="pancreas"),
Example("What is the largest ocean?", expected=["Pacific Ocean", "Pacific"]),
],
scorer=token_f1,
)
pipelines = enumerate_pipelines(registry, task) # every combination
evaluations = evaluate_pipelines(pipelines, executor, eval_set)
best = quality_frontier(evaluations)[0] # highest quality first
best.components, best.quality, best.latency_ms, best.cost, best.success_rate
- Scorers.
token_f1(word overlap, partial credit),contains_answer(the reference appears in the output) andexact_match. Each accepts one reference or a list. Or pass your ownscorer(output, expected)returning a bool or a number from 0 to 1. - Failures. A run that fails part-way scores 0. See When a run fails to tell an outage from a wrong answer.
quality_frontierkeeps the pipelines not beaten on measured cost, latency and quality together.enumerate_pipelineslists every pipeline the task allows;count_pipelinestells you how many that is before you spend any calls.
When a run fails¶
A run can fail because the answer was wrong for the pipeline (a step rejected it) or because the service never answered. Each evaluation records which:
best.errors # (None, "rate_limited", None, ...): one entry per example
best.error_counts # {"rate_limited": 2}
best.quality # failed runs count as 0
best.quality_when_answered # mean over the runs that finished; None if none did
| Kind | Meaning |
|---|---|
rate_limited |
Too many calls; the service refused this one |
timeout |
The call did not finish in time |
unavailable |
The service could not be reached or returned a server error |
not_found |
The model or address does not exist: retired, renamed or misspelt |
auth |
The key is missing, wrong or not allowed to use the model |
empty_output |
The model answered with nothing |
step_failed |
A step returned StepResult(success=False) |
error |
Any other exception, usually a bug in a step |
The first five describe the service, not the pipeline. If a pipeline shows any
of them, run it again before comparing its quality with others: a setup that
was rate limited on half its examples looks half as good as it is.
quality_when_answered shows what it scored on the rest, but it covers fewer
examples, so treat it as a hint and not as the result.
Your own step can name a kind with
StepResult(None, success=False, error="timeout"), and classify_error(exc)
gives the kind of any exception.
Keep the answers, and do not pay twice¶
Wrap any backend in a LoggedBackend and every step is written to a file. A
step that was already answered is read from the file instead of being run:
from frontierkit import LoggedBackend
backend = LoggedBackend(my_backend, "answers.jsonl")
evaluations = evaluate_pipelines(pipelines, AgentExecutor(backend=backend), eval_set)
backend.calls, backend.reused # steps run now, steps read from the file
Run it again and nothing is called. Add a pipeline and only its new steps run.
- What counts as the same step. The same input to the same thing. For a model call that is the endpoint, the model name and the request options, so changing the model asks again. The API key is never written.
- Your own functions. The library cannot see that you changed a function.
Give it a marker and change the marker when the function changes:
LoggedBackend(backend, path, versions={"retriever": "bm25-v2"}). - Failures. A failed call is written with its kind but never reused, so the next run tries it again.
- Several samples.
label="2026-11-21"keeps that day's answers apart from another day's; within one label nothing is asked twice. - What can be logged. Inputs and outputs that are JSON data: text, numbers, lists and dicts. Tuples come back as lists. Anything else still runs, without being logged, and a warning says so.
The file is append-only, one JSON record per line, and it is the trace from a
number in your results back to the call that produced it:
AnswerLog("answers.jsonl").records().
Reporting the best of many¶
The top score among twenty pipelines is flattered: part of it is luck on these
examples. best_of takes that luck out.
from frontierkit import best_of
result = best_of({e.components: e.scores for e in evaluations})
result.winner # the pipeline with the highest score
result.measured # 0.67: its score as measured
result.corrected # 0.50: what to expect on new examples
result.interval # a range that holds although the winner was picked after looking
result.contenders # pipelines the examples cannot tell apart from the winner
The corrected score comes from picking the winner on a random half of the examples and scoring it on the other half, many times over. It is slightly cautious. All pipelines must have been scored on the same examples.
Write the decision down¶
record_decision saves the choice and the evidence for it as one JSON file:
from frontierkit import record_decision
decision = record_decision(evaluations, eval_set, backend=backend, note="for the report")
decision.save("decision.json")
decision.chosen # components, what each step ran on, quality and its range, failures
decision.compared # every other pipeline and how it compares with the chosen one
decision.contenders # the ones too close to call
decision.shared # steps every pipeline ran identically
It picks the highest quality unless you pass chosen=. decision.shared is
the part to quote when two approaches must be shown to have run on the same
model with the same settings. The file also holds a fingerprint of the
examples, the scorer's name and the corrected score of the best pipeline.
Check that the choice still holds¶
A model behind an API can change, slow down or be retired without notice.
verify_decision runs the chosen pipeline again on the same examples and
compares the fresh run with the saved record:
from frontierkit import load_decision, verify_decision
decision = load_decision("decision.json")
check = verify_decision(decision, pipelines, executor, eval_set)
check.status # "valid", "quality_dropped", "slower", "unavailable" or "not_comparable"
check.message # one sentence saying what was found
| Status | Meaning |
|---|---|
valid |
No drop in quality was found, and it is not slower |
quality_dropped |
The same examples now score lower, by more than the tolerance and more than chance could explain |
slower |
Quality held, but most runs take more than twice the recorded time |
unavailable |
The service refused, timed out, or no longer knows the model |
not_comparable |
Other examples, a missing pipeline, or a step that now runs on a different model or endpoint |
Things to know:
- It stops early. Examples are asked one at a time and the check ends as soon as the result is settled, so a clear drop costs a fraction of the calls.
validhas two strengths.check.confirmedis true when the change was shown to be within the tolerance. Otherwise no drop was found with the examples there are, andcheck.detectablesays how big a drop would have gone unnoticed.- Few examples catch only large drops. About 30 examples catch a drop of 0.5, 100 catch 0.3, and 300 catch 0.15, when the model's answers are otherwise stable. If answers vary from run to run, it takes more.
- Tolerance and strictness.
tolerance=0.05is the drop you accept;confidence=0.99is how sure the check must be.slower_factor=2.0andmax_latency_ms=set what counts as slow. - It asks the model again. With a
LoggedBackend, earlier answers are not reused; the fresh ones are written to the same file under their own label. Put theLoggedBackendon the outside of any other backend for this to work.
To run it on a schedule, put what you compared in a Python file:
# compare.py
def setup():
return {"pipelines": pipelines, "executor": executor, "eval_set": eval_set}
frontierkit verify decision.json --setup compare.py
The exit code is 0 for valid, 1 for a quality drop, 3 for slower, 4 for
unavailable and 5 for not comparable, so a scheduled job or CI step fails
when the choice stops holding. Each check is appended to
decision.history.jsonl. Each run is a separate test: at the default
strictness, expect about one false alarm in a hundred runs when nothing has
changed. examples/verify_decision.py shows all of this offline.
Look at each example¶
quality is a mean. Each evaluation also keeps what happened on every
example, in the order of the eval set:
best.scores # (1.0, 0.0, 1.0, ...): one score per example, 0 for a failed run
best.outputs # what the pipeline returned for each example, None for a failed run
Use scores to see which examples a pipeline gets wrong, or to compare two
pipelines example by example. Use outputs to grade the same run with another
scorer without paying for the calls again.
Score extracted triples¶
When a pipeline extracts triples rather than answering a question,
triple_precision, triple_recall and triple_f1 compare what it extracted
with a reference set:
from frontierkit import triple_f1, triple_precision, triple_recall
eval_set = EvalSet(
examples=[Example(document_text, expected=[
("provider", "must document", "training data"),
("user", "may request", "an explanation"),
])],
scorer=triple_f1,
)
evaluation = evaluate_pipelines(pipelines, executor, eval_set)[0]
# Precision and recall for the same run, from the kept outputs:
precision = [triple_precision(output, example.expected)
for output, example in zip(evaluation.outputs, eval_set.examples)]
- The last step must return a collection of triples, each a tuple or list of fields. Text or dicts are rejected, so parse the model's reply first.
- A triple counts only when every field matches after the same normalisation the answer scorers use. Add a field, such as a modality, to make the match stricter.
- Scores are per example, so every example weighs the same however many triples it holds. For figures over all triples together, use metrics.
Figures over the whole set¶
Some figures are not a mean of per-example scores: precision over all triples
in a corpus, or an F1 averaged over labels. Pass those as metrics. Each is a
function of all outputs and all references, computed once per pipeline:
from frontierkit import (
label_macro_f1, pooled_triple_f1, pooled_triple_precision, pooled_triple_recall,
)
eval_set = EvalSet(
examples=examples,
scorer=triple_f1,
metrics={
"precision": pooled_triple_precision,
"recall": pooled_triple_recall,
"f1": pooled_triple_f1,
"modality_f1": label_macro_f1,
},
)
evaluation = evaluate_pipelines(pipelines, executor, eval_set)[0]
evaluation.metrics # {"precision": 0.67, "recall": 0.40, ...}
- Pooled triple metrics count matches over all examples together, so a long document weighs more than a short one. A triple is only matched against the reference of its own example.
label_macro_f1treats one field of each triple as its label (the last by default, orfield=...) and averages the F1 of each label, so a rare label counts as much as a common one. By default a missed or invented triple lowers the F1 of its label, so the figure covers extraction and labelling together. Withmatched_only=Trueonly triples found in both the output and the reference count, so it covers labelling alone:lambda outputs, expecteds: label_macro_f1(outputs, expecteds, matched_only=True).- Your own metric is any
metric(outputs, expecteds)returning a number. The output of a failed run isNone. - Metrics are reported, not searched on: pipelines are still ranked by
quality.
How sure is the result¶
A quality measured on a few examples could easily have come out higher or
lower. mean_interval puts a range around it, and paired_difference says
whether two pipelines can be told apart:
from frontierkit import mean_interval, paired_difference
interval = mean_interval(best.scores)
interval.estimate, interval.low, interval.high # 0.92, 0.55, 1.0
gap = paired_difference(best.scores, other.scores) # same examples, same order
gap.estimate, gap.low, gap.high
gap.verdict # "first is better", "second is better" or "too close to call"
paired_difference compares example by example, which removes the noise from
some examples simply being harder. Two pipelines whose own ranges overlap can
still be told apart this way.
There are two methods:
method |
Use it for | Keep in mind |
|---|---|---|
"betting" (default) |
Scores between 0 and 1, any number of examples | Wider. Stays valid if you recompute it as examples are added, so you may look early and stop |
"bootstrap" |
A narrower range once you have many examples; figures that are not a mean | Approximate, and too narrow on small sets |
On simulated scores, the default range held the true value in at least 97% of 2,000 runs at every size from 12 to 200 examples. The bootstrap range held it in 71% to 94% of runs at 12 examples and about 94% at 30. Expect a wide range from a small set: about 0.5 from end to end at 12 examples, 0.3 at 30 and 0.15 at 200. The range need not be centred on the estimate.
For a metric over the whole set, metric_interval resamples the examples:
from frontierkit import metric_interval
expecteds = [example.expected for example in eval_set.examples]
metric_interval(evaluation.outputs, expecteds, pooled_triple_f1)
- The range reflects the number of examples, not the number of triples. Five long documents are five examples; split them into sections if the sections can be judged on their own.
- These ranges describe one pipeline or one pair. The best of many pipelines looks better than it is, so re-measure the winner on examples that played no part in choosing it.
When there are too many pipelines to try¶
run_closed_loop chooses what to try. Each round it proposes pipelines,
scores the ones it has not scored yet, estimates how good each component is
from the scores, and proposes again.
from frontierkit import ExhaustiveSynthesizer, run_closed_loop
records = run_closed_loop(
registry, task,
synthesizer=lambda reg: ExhaustiveSynthesizer(reg, keep_ties=True),
executor=executor, eval_set=eval_set,
prior_strength=20, iterations=8, explore=2,
)
scored = {e.components: e for r in records for e in r.evaluations}
keep_ties=Truekeeps pipelines that look identical on paper, so each gets measured.explore=Nalso tries the N most promising untried pipelines each round. Without it the loop stops looking once its own shortlist is scored.- A pipeline is never scored twice.
On the project's question-answering benchmark, this finds the best pipeline in
about as many calls as picking pipelines at random. Use it when trying
everything is out of reach; otherwise enumerate_pipelines is simpler and
complete. See Benchmarks.
Search on declared numbers only¶
Without running anything, a synthesizer returns the pipelines that are not beaten on declared cost, latency and reliability:
from frontierkit import BeamSynthesizer, build_demo_registry
registry = build_demo_registry(capabilities=("retrieval", "generation", "verification"))
task = TaskSpecification(
required_capabilities=["retrieval", "generation", "verification"],
max_cost=0.02, max_latency=1500,
)
for pipeline in BeamSynthesizer(registry, beam_width=5).synthesize(task):
print(pipeline.total_cost, pipeline.total_latency, pipeline.total_reliability)
| Synthesizer | Strategy |
|---|---|
ExhaustiveSynthesizer |
Exact frontier; for small spaces |
ProgressiveParetoBeamSynthesizer |
Beam search that widens with depth; the best approximate one |
ParetoPartialBeamSynthesizer |
Beam search with per-stage pruning |
BeamSynthesizer |
Beam search on one weighted score |
HeuristicSynthesizer |
Greedy, one result |
RandomSynthesizer |
Random baseline |
Correct declared numbers from runs¶
calibrate blends what was measured into the registry. With
prior_strength=n, the declared value counts as n observations, so a
component seen a few times barely moves and one seen often converges on what
was measured.
examples/closed_loop.py tests this against a hidden ground truth: the
simulator draws outcomes from true values the loop never sees. Over 20 seeds
and 16 rounds, mean reliability error across twelve components was:
| Calibration | Mean error | Runs that beat the declared values |
|---|---|---|
| None (declared values) | 0.030 | n/a |
Fixed weight, alpha=0.4 |
0.066 | 0 of 20 |
Evidence-weighted, prior_strength=20 |
0.024 | 17 of 20 |
It needs the evidence: at four rounds the evidence-weighted rule does not yet beat the declared values on average.
Save and check registries¶
from frontierkit import load_registry, save_registry, validate_registry_file
save_registry(registry, "registry.json") # writes, then reloads to prove it
validate_registry_file("registry.json").valid
registry = load_registry("registry.json")
frontierkit-registry validate registry.json
Errors name the row and the field. Measured quality estimates are saved with the file.