Can Tests Teach a Prompt to Improve?

Introduction

You change an instruction, run the same task again, and discover that a clearer sentence produced a worse answer. Prompt engineering becomes much more manageable when that discovery is part of an automatic search: a machine proposes alternatives, executes them on cases, measures their outputs, and uses the results to decide what to try next. APE demonstrated this generate–execute–select pattern for natural-language instructions.[1]

So yes: prompts can improve through testing without updating the model that executes them. The useful analogy is test-guided program search. The object being edited is an instruction—or a collection of instructions and examples—while its behavior is measured through the surrounding system. A successful rewrite is a candidate with better observed behavior, not a sentence whose correctness follows from its wording.

Three choices explain much of the difference between optimizers. The search space determines whether a candidate contains only instructions or also demonstrations for several model calls. The proposal mechanism turns examples, score histories or failure traces into new candidates. The selection mechanism determines where subsequent trials are spent. Following those transformations makes it possible to understand both how an optimizer works and what a higher score actually establishes.

What turns wording into an executable candidate?

Start with a task whose outputs can be compared. Suppose an extractor must return an invoice's final payable amount and currency as JSON. Parsing the JSON checks its structure. Comparing the amount with a labeled reference checks a different property. Deciding whether an absent currency should become null is a requirement the evaluator needs before either version can be judged. This is a schematic example, not an experimental result.

The minimal interface is a program that accepts a case and a metric that accepts the expected information plus the actual prediction. DSPy's metric interface takes an example, a prediction and an optional trace, and returns a Boolean or numerical score; its documentation illustrates both exact-answer checks and combinations of semantic assessments with deterministic constraints.[2]

A useful mental model is candidate → executable system → outputs → scores. The score ranks behavior; it need not explain how to repair it. A separate feedback channel can carry an error description, a failed constraint or an intermediate output. Keeping score and feedback distinct lets an optimizer use rich diagnostic information without treating a plausible diagnosis as evidence that the proposed edit works.

For the extractor, “wrong amount” could produce a zero. “You selected the subtotal although a tax-inclusive total was present” gives a reviser a more specific hypothesis. An instruction such as “prefer the explicitly stated final payable total” is then worth trying. Whether it also breaks invoices with only a subtotal remains an execution question.

This distinction also gives the evaluator a concrete job. Use parsers and reference comparisons where the property is executable. Use a semantic judge where interpretation is genuinely needed, then check that judge against independently assessed, representative outputs. Inspect disagreements rather than assuming fluent output means correct output. These are engineering recommendations: the metric interface supplies a place to encode quality, not a guarantee that its author encoded the right quality.

Where does the next candidate come from?

An optimizer needs a way to make a huge space of possible wording searchable. Different methods give the proposing model different information; they do not form a compulsory ladder of increasing sophistication.

APE asks a model to infer plausible instructions from demonstrated input/output pairs, then runs the target model under those instructions and filters candidates using measured scores on training subsets.[1] Here, examples teach the proposer what instruction to write. They need not become few-shot examples included in the final deployed prompt. Its iterative option generates semantically similar variants of surviving instructions, but initial generation and selection already constitute an optimizer.[1]

OPRO supplies a different kind of memory: previous instructions paired with their measured task accuracies, together with task examples and an instruction to propose something better.[3] New instructions are executed and their scores enter the history used by subsequent proposals.[3] The proposing model can look for useful patterns across successful candidates without receiving a written diagnosis of every error. The score history guides generation; it is not a numerical update rule applied to the words.

ProTeGi makes the diagnostic route explicit: execute a prompt on a minibatch—a small group of cases—collect errors, ask a model to describe the prompt's flaws, ask it to edit those flaws, and generate additional paraphrases of the edits.[4] Expansion creates possibilities; beam selection retains a limited set of promising candidates for the next round, while bandit-style evaluation decides which candidates receive further test cases rather than giving each equal evaluation effort.[4] The term “textual gradient” names the criticism-and-editing analogy. It does not make the criticism a derivative or guarantee that the edited prompt improves.

There is also a consequential choice about what counts as a prompt. In a multi-step system, a candidate can specify an instruction and a set of demonstrations for each model-call module. The MIPRO paper bootstraps candidate demonstrations from successful program traces, proposes instructions grounded in program/data context, and searches combinations of these choices across modules.[5] It screens combinations of module instructions and demonstrations on small groups of cases, using a Tree-structured Parzen Estimator to favor promising combinations; periodically, promising combinations are checked on the full optimization set, and the highest-scoring fully evaluated combination is returned.[5]

That changes the optimization problem. Instruction search asks which rule description works; demonstration search asks which worked examples should appear in context; joint search asks which combinations work together. In MIPRO, the demonstrations are part of the selected configuration, rather than merely evidence shown to a proposer. A demonstration accepted because its final program output scored well is still a candidate teaching example—not an independently verified proof of every intermediate step.

How does one edit change a multi-step system?

Consider a system with one model call that writes a search query and another that answers from retrieved documents. A poor answer could result from either step. To revise one instruction usefully, the optimizer needs a representation of the whole candidate, a view of the selected component's behavior, and a way to rerun the system after replacing that component. Follow the update from rebuilding and executing a candidate through scores and selected diagnostic records to proposing and retesting an edit. The next section then follows parent and final-winner selection.

DSPy's GEPA integration makes these objects concrete. For ordinary predictors, compile creates a dictionary mapping predictor names to their instruction strings, passes that candidate plus examples and an adapter into the external optimizer, and reconstructs the returned best candidate as a program.[6] The following is a schematic candidate change, not an actual optimizer output:

Parent candidate                         Child candidate
query_writer: "Find relevant passages"   query_writer: "Search for both entities named in the input question"
answerer:     "Answer from passages"     answerer:     "Answer from passages"

The unchanged answerer entry matters. The candidate is a complete assignment, even when the mutation changes only query_writer. The adapter deep-copies the supplied program and applies matching instruction entries to its predictors.[7] This is the decisive instruction-update loop from build_program; code-rebinding logic earlier in that function is omitted because this example concerns ordinary predictors, not source-code optimization.[7]

for name, pred in new_prog.named_predictors():
    if name in candidate:
        pred.signature = pred.signature.with_instructions(candidate[name])

The dictionary is therefore not the model's final request or an execution log. It is the configuration used to rebuild an executable program. Running that program produces predictions and, when requested, traces. In the ordinary-predictor path, the adapter's traced evaluation invokes program execution with failed-parse capture enabled, then packages outputs, numerical scores and trajectories for the optimizer.[7]

Reflection consumes a more focused object. For a selected predictor, make_reflective_dataset finds matching invocations in the trace and builds records with Inputs, Generated Outputs and Feedback; when several valid invocations are available, it selects one, preferring a failed prediction when such failures are enabled.[7] The feedback callback receives both predictor-level inputs/outputs and the whole program's input, output and captured trace.[6] Thus the callback can diagnose a query-writing mistake in the context of the final answer, while the instruction proposer receives constructed component records rather than necessarily every byte of the original execution.[7]

Two implementation choices change what can be learned. Failed parses can be captured during execution yet filtered out of reflective examples: add_format_failure_as_feedback defaults to False.[6, 7] Also, feedback text can target a predictor, but if its returned numerical score differs from the program-level score, the adapter replaces it with the program-level score.[7] Localized criticism and independently scored local optimization are different things. The former is available here; the latter is not established by that callback.

The GEPA paper describes the resulting mutation cycle: choose a parent and a module, run a feedback minibatch, revise that module's instruction using traces and feedback, and rerun the updated system on the same minibatch.[8] Its algorithm sends an improved child on to evaluation over the larger selection set.[8] The local edit is therefore tested through its downstream consequences. A better-looking search instruction earns no acceptance merely by looking better.

Which candidates deserve another chance?

A score is useful for more than choosing the final winner. It determines which candidates deserve additional evaluation and which should become parents of later edits. APE filters candidates before spending further scoring effort; ProTeGi maintains a beam of promising prompts.[1, 4] These choices allocate the search budget as well as rank the results.

GEPA's paper uses a different parent-selection rule. It first forms a pool of candidates that attain the best score on at least one selection-set case, prunes that pool, and samples among the remaining case winners with weight proportional to how many cases each leads.[8] A candidate need not have the highest overall average to remain a useful parent.

Imagine, purely as an illustration, one instruction that handles ordinary invoices well and another that is unusually good at missing-currency cases. An average-score-only search might repeatedly edit the first. Keeping the second as a parent preserves a different place to search for useful rules. This is diversity in candidate behavior, measured on cases—not diversity inferred from how different two instruction strings sound.

Parent selection and deployment selection are separate. GEPA's paper returns the candidate with the highest aggregate score on its selection set after the budget is exhausted.[8] It does not deploy an oracle that chooses each future input's historically best prompt. Nor does retaining complementary candidates guarantee that combining their wording will help: in the paper's Qwen3-8B IFBench experiment, the initial system scored 36.90, GEPA 38.61, and GEPA+Merge 28.23.[8]

The paper's selection ablation compares outcomes under different selection policies within a fixed reflective harness: across HotpotQA, IFBench, HoVer and PUPA with Qwen3-8B, aggregate improvement over the initial system was 12.44 percentage points with GEPA's selection, 6.05 with current-best selection, and 5.11 with beam search.[8] The algorithm gives a rationale for spreading search across case winners; the ablation measures the selection policy's outcome, not diversity as an independently identified cause. It does not isolate detailed reflection from generic rewriting, and it is not a universal ranking of all optimizers called “beam search.”

The proposal mechanism needs its own test. A study using Claude 3.7 Sonnet on GPQA Diamond, Web of Lies and Multistep Arithmetic found that gradient-like, multi-step feedback processing did not consistently outperform direct one-step feedback-conditioned rewriting; Web of Lies under the study's named “naive selection” condition was an exception favoring the gradient-like structure.[9] The same study found benefits from asking for improvement without task feedback in some settings, but feedback-driven variants won several comparisons, and its validated comparisons had limited statistical power.[9] The practical inference is to test the added mechanism against a simpler proposer, not to assume either that reflection is necessary or that feedback is useless.

What improvement can the loop actually deliver?

Those comparisons ask which proposal or selection mechanism works better within an optimization setup. A separate question is whether the selected prompt improves performance on final test cases kept outside both editing and winner selection. In MIPRO's Llama-3-8B experiments, test scores on ordinary HotPotQA were 36.1 for the initial system, 36.8 for instruction-only MIPRO, 45.8 for demonstration-only Bootstrap Random Search, and 46.4 for joint MIPRO, averaged across five optimizer runs.[5] On its rule-rich HotPotQA Conditional task, the corresponding scores were 6.0, 14.6, 10.4 and 23.3.[5] Selecting demonstrations was a strong alternative in the ordinary task; the conditional task gave instruction optimization a more consequential role. These are results from that model and experimental setup, not a permanent rule that examples beat instructions.

GEPA's Qwen3-8B study gives a newer, differently configured comparison: HotpotQA test scores rose from 42.33 initially to 62.33 with GEPA, while the weight-updating GRPO comparator scored 43.33; on AIME-2025, GEPA rose from 27.33 to 32.00, below GRPO's 38.00.[5, 8] The reported GEPA optimization budgets were 6,871 rollouts for HotpotQA and 1,839 for AIME, versus 24,000 for each GRPO run; a rollout here means invoking the task system and evaluating its output, rather than necessarily making a single model call.[8] HotpotQA used 150 training, 300 validation and 300 test examples; AIME used 90 earlier-year problems split between training and validation, with 30 AIME-2025 test problems generated five times each.[8] Those five generations are not five independent searches. Weight updating changes a different artifact, so its results are a comparator, not a step required for prompt optimization.

To build a usable loop, turn these mechanisms into explicit choices:

1. Define the task and the editable configuration. Decide whether the search may change instructions, demonstrations, or both, and which modules are in scope. Keep the initial system as a baseline.
2. Build scoring before search. Run baseline cases, retain outputs and failure details, and check that success means the intended behavior. Make critical failures separately visible; a low per-case score can still be outweighed in an average.
3. Give proposals the information they need. Use demonstrated behavior for instruction induction, score history for history-conditioned generation, or component inputs/outputs and feedback for targeted revision. Execute proposed configurations rather than asking the proposer to certify them.
4. Spend trials deliberately. Choose how to screen candidates, retain parents and select a final configuration. Compare a simple rewrite or demonstration-search baseline with richer reflection under a declared budget; an equal number of task evaluations need not mean equal total cost.
5. Estimate new-case behavior only after selection. Keep a final test outside both editing and winner selection, compare old and new systems on the same cases, and inspect the important failure groups as well as the mean.

This is a recommended assembly of the demonstrated operations, not one universally validated recipe. DSPy's optimization guidance explicitly calls for a held-out test beyond development and optimization data, and warns about overfitting small sets.[10] A validation set used repeatedly to choose parents or winners participates in search even if its text never reaches the rewriting model. Define “new” according to the intended application: different document families, customers, topics or time periods may require grouped separation. Repeat executions—and complete searches where search randomness matters—to expose uncertainty consequential to the decision; no fixed split or repetition count follows for every task.

Finally, inspect what the errors ask you to change. Missing source information calls for a data or retrieval remedy; contradictory labels call for a specification decision; an inadequate operation may call for a tool or program change. These are diagnostic judgments, not automatic prescriptions inferred from a failed search. DSPy's guidance likewise invites developers to revisit the task, data, metric and program structure.[10] People accountable for the application still authorize the objectives, permitted changes and release conditions, even when software helps formulate or enforce them.

Conclusion

Tests can drive automatic prompt improvement because an optimizer can turn measured behavior into new search decisions. Examples can suggest instructions, score histories can guide proposals, traces can support targeted revisions, and per-case results can preserve useful alternatives. The final candidate must still run through the system it is supposed to improve.

The strongest version of the program analogy is therefore operational: define an editable artifact, execute it, measure it, and search again. Understanding which information crosses each boundary is more useful than treating every rewrite as learning—or every “textual gradient” as mathematics.

Limitations

The invoice and two-module examples above are schematic. Paper results establish behavior in their reported settings, while the DSPy walkthrough establishes static implementation paths at the referenced revision. It does not establish that the separately released optimizer and wrapper were installed and exercised together. The paper describes its search and selection algorithm; the walkthrough inspects DSPy's wrapper and does not establish every behavior of the external optimizer it calls. Source-code optimization through DSPy's Flex modules lies outside the ordinary-instruction path explained here.

Finite task scores do not prove general correctness, acceptable minority-case behavior or reliable performance after execution conditions change. Positive transfer is possible: GEPA reports gains when Qwen3-8B-optimized prompts are applied unchanged to GPT-4.1 Mini.[8] That experiment does not establish gains after a future snapshot update of the same deployed model. Treat a change of model, decoding, retrieval, tools or parser as grounds to rerun representative cases and critical checks, rather than assuming text portability guarantees behavioral portability.

Rollout counts also do not establish proportional monetary savings. GEPA's paper reports total GPT-4.1 Mini optimization-experiment costs of $86 for GEPA and $76 for MIPROv2, despite GEPA's higher aggregate task performance in that comparison.[8] Those totals are research-search costs, not deployment returns or current price quotations. A deployment decision additionally needs proposal/judge overhead, final prompt and output length, reuse volume, and the value or harm of changed behavior. Neither a universal optimizer winner nor a break-even threshold follows from the reported studies.