How to pick a model for an AI worker
Benchmarks measure somebody else's task. The only comparison that counts is the same run, on your instruction and your data, judged by you.
A kit brings the instructions, not the model. Claude, ChatGPT, Gemini and Grok all drive the same kits, so choosing between them is a decision you make at deploy time and can revisit whenever you like.
Which raises the obvious question: on what basis?
Benchmarks answer a different question
Public benchmarks measure performance on somebody else's task, with somebody else's prompt, scored by somebody else's rubric. That is genuinely useful for model builders and close to useless for choosing a model for one narrow job.
| A public benchmark | Your worker's job | |
|---|---|---|
| The task | Standardised, general | One specific job |
| The prompt | The benchmark's | Your instruction |
| The data | Public, curated | Your inbox, your CRM |
| What counts as right | A rubric | Your judgement |
| Failure that matters | Average score | The specific way it gets your job wrong |
A model can lead a reasoning leaderboard and still be the wrong pick for summarising your meeting notes, because your notes have a house shorthand it keeps misreading. No benchmark contains that.
The comparison that counts
Send the same run to a different model and read both outputs.
That is the whole method, and it works because it removes every variable except the one you are testing: same instruction, same inputs, same day, same job. You are not comparing models, you are comparing this model on this job.
Meeting Prep Assistant is a good stress test because it reaches four apps (calendar, email, meeting notes, CRM) and has to synthesise rather than extract. Run it twice on two models and read both prep sheets. The differences are usually obvious within a morning: one attributes a comment to the wrong attendee, one buries the open item, one is simply longer without being more useful.
That is evidence. A leaderboard position is not.
Reading a per-model grade
Every kit page grades itself per model, and each grade carries its provenance:
| Provenance | What it is | How much weight |
|---|---|---|
| Measured | Taken from real runs of this kit | Evidence |
| Declared | Stated by the kit's publisher | A claim worth checking |
The mark is the useful part. A measured grade means somebody ran it; a declared one means somebody believes it. Both are worth having, and they deserve different confidence before you put a worker on a job that touches customers.
You can also filter the directory by model to see only kits that fit the one you already use.
What actually varies between models
In practice, for unattended work, the differences that show up are narrower than the discourse suggests:
- Instruction adherence over long context. Does it still respect the rule on run four hundred, with an unusual input?
- Restraint. Does it stop when it should, or invent a plausible answer to fill a gap?
- Formatting stability. A worker whose output another system reads needs the shape to be the same every time.
- Cost per run, which for a worker reading long threads is dominated by input tokens rather than by cleverness.
Notice that three of those four are about consistency, not capability. Unattended work rewards predictability in a way chat does not, because nobody is there to notice an off day and rephrase.
What it costs to run the comparison
Testing is cheap, which is the argument for doing it properly.
A single comparison run of a multi-app worker costs roughly what one normal run costs. Using tool calls as the unit:
| Test | Runs | Tool calls | Against Free's 500 a day |
|---|---|---|---|
| One job, two models | 2 | ~40 | 8% |
| One job, four models | 4 | ~80 | 16% |
| Three jobs, two models | 6 | ~120 | 24% |
Model tokens are billed separately at the **provider's list price with no markup**, from the wallet or your own key. Prices vary by provider and change, so check the provider's own pricing rather than trusting a number in a blog post, including this one.
The no-markup part is what makes this honest advice rather than a sales pitch: WorkerKit earns nothing extra from you picking the expensive model, so there is nothing steering the recommendation.
The failure mode: choosing once, by reputation
The common mistake is picking the model with the best reputation, deploying every worker on it, and never revisiting.
It fails in two directions. Your triage worker is probably over-provisioned, paying frontier prices to classify email into four buckets. And your genuinely hard job may be on a model that is no longer the best at it, because the leaderboard moved in March and nobody re-checked.
The fix is not constant churn. It is picking per job rather than per account, and re-checking when there is a reason: a new model ships, a worker starts producing worse output, or the token bill grows faster than the workload.
The reasonable objection
"Running every job on multiple models and judging outputs by hand does not scale. I have twelve workers."
Correct, and the answer is that you do not do it for all twelve. Do it for the two or three where quality actually varies and the output reaches a customer. For the rest, pick a sensible default and rely on the per-model grades.
The other honest limitation: your judgement is the rubric, which means the comparison is only as good as your ability to tell a good prep sheet from a convincing one. That is a real constraint, and it argues for comparing on jobs where you know the domain well rather than on the ones you were hoping to stop thinking about.
When to pin a model and leave it
- The job is measured and working. Novelty is not a reason to re-qualify a worker that touches customers.
- Compliance constrains the choice. If one provider is approved, that is not yours to optimise.
- Output feeds another system. Format stability matters more than marginal quality; changing models means re-checking the contract.
- Cost is already trivial. A once-daily worker's token spend rarely justifies the attention.
Being able to move cheaply is the value. Moving constantly is not.
FAQ
How do I choose which AI model to run a worker on?
Compare on your own job rather than on benchmarks: send the same run to two models with the same instruction and the same inputs, and read both outputs. That removes every variable except the model, which is the only comparison that predicts how it will behave on your work.
What does a per-model grade on a kit page mean?
It scores how that kit performs on that model, marked with where it came from: measured from real runs, or declared by the publisher. Treat a measured grade as evidence and a declared one as a claim worth checking before a worker touches customers.
Can I change a worker's model after deploying it?
Yes, at any time and with no rewrite, because a kit brings the instructions rather than the model. You can also send a single run to a different model to compare them on the same job.
Does running a more expensive model cost more on WorkerKit?
You pay the provider's list price for its tokens, with no markup added, so a more capable model costs what that provider charges and nothing extra. Check the provider's own pricing for current rates. See /pricing.
Should every worker use the same model?
Usually not. Pick per job: a classification worker rarely needs a frontier model, while a synthesis job across four apps might. Choosing per account tends to over-provision the simple workers and under-serve the hard one.