AI evaluation budget worksheet
Estimate model-call cost and human review time before comparing models. Inputs remain on this page and are not submitted or saved. Starting prices are examples, not current vendor quotes.
Estimate for one model
300 calls · $1.20 token cost · 10.0 review hours
Worked example
100 cases repeated three times produce 300 calls. At 2,000 input tokens and 500 output tokens per call, example prices of $1 and $4 per million give $1.20 in token cost. Two minutes of review for each run adds up to 600 minutes, or 10 hours. Human evaluation can dominate the effort even when API use looks inexpensive.
What the estimate leaves out
The formula is calls × (input tokens × input rate + output tokens × output rate) ÷ 1,000,000. It excludes tool calls, retrieval, retries, caching discounts, hosting, taxes and reviewer pay. It assumes the same token usage on every call. Record actual usage after a pilot and replace the averages. Compare multiple candidates separately if their token usage differs.
Design the evaluation before running it
- Choose real tasks, including ambiguous instructions and missing information. Include cases where stopping or asking a question is correct. Remove private data before sending examples to a service.
- Define success before seeing outputs. For extraction, score format and factual field accuracy separately. For agents, check the resulting action, not only the explanation.
- Keep inputs and available tools consistent between candidates. Record model version, date, parameters and retry rules. Include failed runs in the denominator.
- Repeat cases to expose variability. Three runs here are a budgeting example, not a claim of statistical sufficiency. A test-set pass rate does not guarantee future performance.
- Separate severe failures from minor errors. An unauthorized action should not disappear inside an average score dominated by easy cases.
Keep a decision record
Record the task, model version, source of cases, pass criteria, severe failures, latency, actual cost, reviewer disagreements and unresolved limitations. Use those observations to decide whether to roll out. This worksheet estimates effort; it does not rank models or claim hands-on benchmark results.