How FrontierKit works¶
FrontierKit answers one question: of the setups you could use, which one should you use, and how sure can you be? A setup is a pipeline: one choice for each step, such as a way of finding context, a prompt and a model.
It gets there in stages. You describe your questions and your options, it runs every pipeline on every question, scores the answers, works out what the scores can and cannot tell apart, and keeps the pipelines nothing beats. The choice can be written down and checked again later.
flowchart TD
questions["<b>Your questions</b><br>examples, a scorer, reference answers"]
options["<b>Your options</b><br>components in a registry, steps in a task"]
pipelines["<b>Pipelines</b><br>every combination, or a search"]
execution["<b>Execution</b><br>your functions and any model vendor"]
answers["<b>Answer log</b><br>every reply kept, none paid for twice"]
evaluation["<b>Evaluation</b><br>a score per example, failures named"]
statistics["<b>Statistics</b><br>ranges, paired comparisons, best of many"]
frontier["<b>Frontier</b><br>the pipelines nothing beats"]
decision["<b>Decision record</b><br>what was chosen, and on what evidence"]
recheck["<b>Verify</b><br>the chosen pipeline, run again weeks later"]
options --> pipelines --> execution --> evaluation --> statistics --> frontier --> decision --> recheck
questions --> evaluation
execution <--> answers
What you describe¶
- A component is one option for one step: a model, a prompt template, a
retriever.
ComponentMetadatadescribes it, with the step it can fill and starting guesses for its cost, speed and reliability. - A registry (
CapabilityRegistry) holds the components. It can be saved to and loaded from a JSON file, andfrontierkit-registry validatechecks such a file. - A task (
TaskSpecification) names the steps a pipeline needs, in order, with optional limits on cost, latency and privacy. - An eval set (
EvalSet) holds your examples and how to grade an answer.
The declared numbers are only starting guesses. Everything that follows is measured.
Pipelines¶
A pipeline is one component per step, run as a straight chain (AgentGraph).
enumerate_pipelineslists every pipeline the task allows, andcount_pipelinessays how many that is before any call is made. Trying them all is the dependable way.- When there are too many to try,
run_closed_loopchooses which to score next from the scores it has seen. On the project's benchmark it currently finds the best pipeline in about as many calls as picking at random. - The synthesizers (
ExhaustiveSynthesizer,ProgressiveParetoBeamSynthesizerand others) search on declared numbers alone, without running anything. Benchmarks compares them.
Execution¶
AgentExecutor runs a pipeline step by step, handing each step's output to the
next. A backend decides how a step runs:
| Backend | What it runs |
|---|---|
CallableBackend |
Your own Python functions |
OpenAIBackend |
Any vendor or local server that accepts OpenAI-style chat requests |
RoutedBackend |
A different backend for each component, to mix vendors and functions |
LoggedBackend |
Any of the above, with every answer written to an AnswerLog |
Each step is timed and costed, and a pipeline stops at its first failed step. A
failure is recorded as what it was: rate_limited, timeout, unavailable,
not_found, auth, empty_output, step_failed or error.
With an answer log, a step that was already answered is read back from the file instead of being called again. What makes two calls the same is what ran (the endpoint, model and request options, or a version you give your own function) and what it was asked.
Without a backend the executor simulates: steps succeed always, or with the probability their declared reliability gives. That mode exists to test search strategies and produces no answers to score.
Evaluation¶
evaluate_pipelines runs every pipeline on every example and returns a
PipelineEvaluation for each:
quality: the mean score, counting a failed run as 0.scores,outputs,latenciesanderrors: what happened on each example.cost,latency_msandsuccess_rate: measured, not declared.metrics: figures over the whole set, such as precision pooled over a corpus.
Scorers grade one answer against its reference. exact_match,
contains_answer and token_f1 cover short answers, and triple_precision,
triple_recall and triple_f1 cover extracted triples. Any function returning
a number from 0 to 1 works.
Statistics¶
A score from a few dozen examples could easily have come out higher or lower.
mean_intervalgives the range a quality plausibly lies in.paired_differencecompares two pipelines example by example and answers "first is better", "second is better" or "too close to call".best_ofreports the winner among many with the luck of the pick taken out, and lists the pipelines that cannot be told apart from it.metric_intervalgives a range for a figure over the whole set.
The default method only assumes scores stay between known bounds. It is valid for any number of examples and stays valid when you look repeatedly as examples arrive. See How sure is the result.
Frontier¶
quality_frontier keeps the pipelines that are not beaten. One pipeline beats
another when it is at least as good on measured quality, cost and latency, and
strictly better on one of them. What is left is the set of real trade-offs; the
choice among them is yours.
Decisions that can be checked¶
record_decision writes down which pipeline was chosen, what each step ran on,
the range of every score, each pipeline compared with the chosen one, and the
failures by kind. verify_decision, or frontierkit verify on the command
line, runs the chosen pipeline again on the same examples and answers:
| Status | Meaning |
|---|---|
valid |
No drop in quality was found, and it is not slower |
quality_dropped |
The same examples now score lower, by more than chance explains |
slower |
Quality held, but most runs take much longer |
unavailable |
The service refused, timed out, or no longer knows the model |
not_comparable |
Other examples, or a step that now runs on something else |
Declared numbers and measured ones¶
calibrate blends what runs recorded into a new registry: a component seen a
few times barely moves from its declared values, and one seen often converges
on what was measured. calibrate_quality does the same for answer quality.
run_closed_loop uses both between rounds.
Where the code lives¶
| Package | Holds |
|---|---|
frontierkit.registry |
Components, registries and registry files |
frontierkit.synthesis |
Tasks, enumeration and the search strategies |
frontierkit.graph |
A pipeline as a chain of components |
frontierkit.execution, frontierkit.adapters |
The executor, backends and failure kinds |
frontierkit.store |
The answer log |
frontierkit.evaluation |
Eval sets, scorers and whole-set metrics |
frontierkit.stats |
Ranges, paired comparisons and best of many |
frontierkit.optimization |
Which pipelines beat which |
frontierkit.telemetry, frontierkit.loop |
Calibration and the closed loop |
frontierkit.decision, frontierkit.verify, frontierkit.cli |
Decision records and checking them |
Limits¶
- Pipelines are straight chains of steps. Branching pipelines are not searched.
- A few dozen examples can only separate large differences. The results say what they cannot tell apart.