How much accuracy can you afford?

A more expensive model does not automatically buy a better result on your task. And a longer, more explicit prompt does not automatically save money.

In this small healthcare keyword experiment, an optimized prompt running on DeepSeek V4 Flash passed the same 20 evaluation cases as the original prompt running on GPT-5.6 Sol. Its estimated serving cost was about one-tenth as much.

That is a useful lead, not proof of general equivalence. The cases were exploratory, their held-out status was unverified, and the rubric measured a narrow set of requirements. The experiment nevertheless shows how to connect Prompt Studio to Benchmarking and ask a practical question: how much does the quality you need actually cost?

Define the task and the dynamic input

Our task is to suggest search marketing keywords for a healthcare chain. The service changes with each invocation; the instructions stay fixed.

In Prompt Studio, the task description was:

Search marketing keywords exploration for a healthcare chain.

The original system prompt was:

Search marketing keywords exploration for a healthcare chain. Your goal is to identify new prospective keywords to generate new user demand for the healthcare clinics owned by the chain.

3 keywords in total.
Each keyword must contain less than or equal to 5 words.
Each keyword must be relevant to the healthcare domain.

The user prompt was:

Suggest 3 keywords for {{service}}.

The double braces mark a template variable. Supplying the following input:

{"service": "physical therapy"}

produces this user message:

Suggest 3 keywords for physical therapy.

A later invocation can supply a different service without rewriting the prompt-set. Keeping the variable in the template also lets the benchmark apply the same test inputs to both prompt versions.

Optimize the prompt-set in Prompt Studio

This run selected GPT-5.6 Sol as the target model and Claude Opus 4.8 as the reflection model that rewrote the prompts. It used zero-shot optimization: a rewrite without a scored training or validation loop. The optimization record contains no before-and-after quality scores. We establish quality in the benchmark that follows.

The rewrite made the output format explicit, repeated the constraints in the user prompt, and added guidance about patient search intent. It also introduced a dermatology example.

An excerpt from the optimized system prompt illustrates the change:

OUTPUT REQUIREMENTS:
1. Return exactly 3 keywords.
2. Each keyword must contain 5 words or fewer.
3. Each keyword must be clearly relevant to the healthcare domain.
4. Prefer natural, realistic search queries that reflect genuine patient intent (symptoms, conditions, treatments, or services).
5. Avoid duplicates, overly generic single-word terms, and keywords unrelated to healthcare.

FORMAT:
- Present the 3 keywords as a numbered list (1, 2, 3), with one keyword per line and no extra commentary.

The optimized user prompt preserved the dynamic input:

Suggest exactly 3 search marketing keywords for the following healthcare service: {{service}}.

Follow these rules:
- Each keyword must be 5 words or fewer.
- Each keyword must be directly relevant to {{service}} and the healthcare domain.
- Focus on terms real patients would search when looking for this service (e.g., conditions, treatments, locations, or intent phrases).

Return the result as a numbered list:
1. <keyword>
2. <keyword>
3. <keyword>

These additions are hypotheses about better instructions. They still need testing, including on models other than the selected target.

Derive QCR from the original instructions

Before running the benchmark, define what counts as success. Benchmarking offers Draft from prompt, which drafts the Quality Criteria Rubric, or QCR, from the baseline system and user prompts. Each proposed criterion cites its source text for review.

The optimized prompt does not participate in drafting the rubric. Optimization can change how we ask; it cannot redefine what counts as success.

This benchmark used three criteria, all stated in the baseline:

The judge was Claude Opus 4-8 Fast, recorded as claude-opus-4-8-fast, and was separate from the three contenders.

A response passes only when all three criteria pass. The report also stores partial-credit scores, but the percentages below are the proportion of responses passing the complete rubric.

This definition matters. Healthcare relevance and keyword length do not establish search volume, conversion potential, or business value. The optimized prompt adds preferences about intent and duplication, but those preferences do not silently become new benchmark requirements.

Run both prompt arms across the contenders

The benchmark used GPT-5.6 Sol, DeepSeek V4 Flash, and Claude Sonnet 5.5. Each received the original and optimized prompt-sets with the same 20 service inputs, producing 120 task responses.

Model names follow the IDs recorded in the saved run. Its manifest, result arms, and billing receipts identify the Sonnet contender as claude-sonnet-5.5. The saved price snapshot assigns that ID $1.40 per million input tokens and $7 per million output tokens. See the benchmark source summary for the recorded model IDs, rates, and aggregate results.

The dataset was generated from seed examples and included physical therapy, cardiology, sleep medicine, and other healthcare services. It contains no exact duplicate inputs.

A generated dataset is not automatically a held-out test set. This run records its holdout status as unverified; the generator checked seed overlap but lacked the complete optimization and selection history. Dermatology also appears both in the generated cases and as an example in the optimized prompt. These results therefore belong to an exploratory evaluation.

For a final held-out benchmark, freeze the prompts, rubric, and contender list first. Generate cases that exclude all optimization examples and previously used validation or selection cases, review semantic overlap, and keep the final set untouched until evaluation.

Compare quality and serving cost

All 120 responses received usable rubric evaluations in this run.

Costs use measured token usage and the benchmark’s saved price snapshot, extrapolated to 1000 requests. They describe serving the task; they exclude optimization, dataset generation, and evaluation costs. They are estimates from this run, not current price quotes.

GPT’s optimized prompt increased estimated serving cost by about 38% without changing its observed pass rate. DeepSeek’s optimized prompt increased cost by about 25% and improved from 19 to 20 passing cases. Its optimized arm cost about 90% less than GPT’s baseline arm.

With just 20 cases, one outcome changes the pass rate by five percentage points. A 20-of-20 result has an approximately 84% to 100% reported 95% interval. Matching observed scores does not establish statistical equivalence.

Inspect the answers behind the scores

Sonnet’s result needs particular attention. Its baseline often returned a keyword followed by an explanation, such as:

1. **post-surgery rehab near me**: targets patients who need recovery support after procedures like knee or hip replacements and are searching for a nearby clinic.

The length checker counted the entire list item, including the explanation, against the five-word limit. The optimized prompt explicitly requested keywords without commentary.

That makes output formatting a substantial part of the measured improvement. It does not show that baseline Sonnet was incapable of suggesting relevant keywords. Before using the pass-rate jump as a headline, the evaluator must distinguish the intended keyword from accompanying prose consistently with the baseline’s requirements.

The token measurements also explain why more detailed instructions can sometimes lower the bill:

All three received more input tokens. Only Sonnet’s output reduction was large enough to lower its estimated serving cost in this run. Prompt length alone is an incomplete cost measure.

Read the quality and cost frontier

Plot estimated serving cost on the horizontal axis and all-criteria QCR pass rate on the vertical axis. Each point represents a model and prompt arm. A configuration is dominated when another observed configuration is at least as accurate and no more expensive, with an improvement in at least one dimension.

Using the observed point estimates, the frontier contains two configurations:

The line connecting these points shows the observed tradeoff between two discrete choices. Intermediate points do not represent configurations we tested.

The optimized DeepSeek configuration has an estimated serving cost about $0.0165 higher per 1000 requests. In the 20-case benchmark, it passed 20 cases compared with the baseline’s 19, an observed difference of five percentage points. This small sample does not establish an expected improvement on future requests. All other configurations are dominated on the two plotted measures. This is an empirical frontier for this experiment, with uncertainty in the quality estimates, rather than a universal ranking of the models.

Choose the accuracy you can afford

Start with the quality requirements your application actually needs. Then compare the cost of meeting them across prompt-sets and models.

This experiment gives us a promising candidate: a cheaper model with the optimized prompt matched the target model’s observed pass rate at roughly one-tenth the estimated serving cost. It also gives us reasons to test again: a small exploratory dataset, an overlapping prompt example, and a length checker whose treatment of explanatory text affects the comparison.

A fresh held-out evaluation, with the scoring interpretation settled before execution, is the next test of that candidate. If the advantage survives, we have evidence for a deployment choice.

The useful question is not simply whether optimization improved a prompt. It is which prompt-set and model meet the quality bar at a cost we can afford.