Evaluate tool use before changing your agent architecture

Choose the question first

Anthropic describes tool search as loading relevant tool definitions on demand, and programmatic tool calling as running orchestration code while keeping intermediate results outside the model context. Its MCP engineering article describes a related pattern of processing tool data in a code execution environment. These are different interventions: finding a tool and processing its output can fail independently. Advanced tool use; Code execution with MCP.

Our suggested experiment: start with direct calls as the baseline. Add discovery in a second configuration. Add code orchestration in a third. Keep the model, permissions, input data, retry allowance and output requirements fixed. Changing all of them together makes a better result difficult to explain.

Make a small, checkable task set

For a fictional support assistant, prepare 20 cases: eight straightforward record lookups, four ambiguous names, four missing records, and four requests that must be refused because the account lacks permission. Give each case an expected record identifier and allowed action. Use synthetic records, so the test can be repeated without exposing customer information.

Define success before running: the correct identifier, every required answer field, and no forbidden action. A valid refusal counts as success when refusal is expected. A guessed answer for a missing record fails even if its prose looks convincing. Preserve individual traces so a reviewer can inspect the evidence behind the answer.

Calculate cost per successful task

Suppose configuration A costs $0.80 across 20 attempts and succeeds on 16. Its cost per successful task is $0.80 ÷ 16 = $0.05. Configuration B costs $0.60 and succeeds on 12, giving the same $0.05. B is cheaper per attempt but delivers fewer completed tasks. These numbers are invented to demonstrate the calculation.

Include retry charges and execution fees in the numerator. Record human review minutes separately; do not hide them inside a token metric. When no task succeeds, report cost per success as undefined and keep the total spend visible. Use the evaluation budget worksheet to estimate the run budget before testing.

Record failures before picking a winner

Use one row per attempt with case ID, configuration, repeat number, success, elapsed seconds, total charge, review minutes, tool selected and failure reason. Separate discovery failures from incorrect arguments, data-processing errors and unsupported final claims. An aggregate success rate cannot tell you which component to fix.

Repeat each case three times and compare configurations on the same cases. Report successes over attempts, plus which cases failed consistently. Sixty correlated attempts are not sixty independent customer scenarios. This small exercise finds obvious regressions; it does not establish a universal accuracy claim.

Make the release decision explicit

Set your acceptable correctness, latency and review burden before looking at the results. Treat a permission violation as a separate release blocker rather than averaging it away with easy successes. If the outcome is mixed, keep the baseline and add cases that distinguish the failure modes. Publish the task definitions, assumptions and limitations alongside any claimed improvement so someone else can interpret it.