The prompt
Score 10 of 10, Excellent
Use it when A test passes and fails at random, and you want the cause fixed rather than the test retried, skipped or loosened.
- {{test_name}}
- {{test_file}}
- {{failure_pattern}}
- {{failure_output}}
- {{single_test_command}}
- {{suite_command}}
- {{off_limits}}
Full prompt and why it works
Prompt
Goal: Make the flaky test {{test_name}} pass reliably by fixing the cause of the flakiness, whether it is in the test or in the code under test. Context: - Test file: {{test_file}} (for example src/cart/checkout.test.ts) - How often it fails, and where (locally, in CI, only in parallel runs): {{failure_pattern}} - Output from a failed run: """ {{failure_output}} """ - Command to run just this test: {{single_test_command}} - Full suite command: {{suite_command}} Constraints: - Reproduce the failure first. Run the test repeatedly, for example 50 times in a loop, and also together with the rest of its file in random order if the runner supports it, until you see it fail. If it only fails in CI, match the CI conditions you can see in the config, such as the number of parallel workers and the environment variables. Report the failure rate. - Find the cause before editing and explain it in 2 to 3 sentences. Causes to check include timing and sleeps, state shared between tests, test order, the real clock and time zones, randomness without a fixed seed, network or external services, async work that is not awaited, and the order of items in unordered collections. - Fix the cause. Do not add retries, raise timeouts as the fix, mark the test as skipped or flaky, or weaken its assertions, because those hide the problem rather than solve it. - If the cause is a real bug in the code under test, such as a race condition, stop and describe it before changing that code. - Do not touch {{off_limits}}, and do not change other tests unless they share the state that causes the problem; say so if they do. Done when: - The test passes 100 times in a row using the same loop you used to reproduce it. - The full suite passes: {{suite_command}} Verify: run both, then report in this format: Cause (2 to 3 sentences), Fix, Failure rate before, Pass count after, Commands run, and the last 20 lines of their output. If you cannot reproduce the failure after 200 runs, stop and tell me what you tried rather than changing code on a guess.
Variables
- {{test_name}}
- The test's name as the runner shows it, for example "checkout applies discount once".
- {{test_file}}
- The file it lives in, for example src/cart/checkout.test.ts or tests/test_checkout.py.
- {{failure_pattern}}
- How often and where it fails, for example "about 1 in 20 CI runs, never locally".
- {{failure_output}}
- The exact output from a failed run, pasted rather than described.
- {{single_test_command}}
- The command to run only this test, for example npx jest src/cart/checkout.test.ts -t "applies discount".
- {{suite_command}}
- The command for the full test suite.
- {{off_limits}}
- Files, folders or systems that must not change, for example the CI configuration.
Why it works
- What to avoid. Retries, longer timeouts, skips and looser assertions are ruled out, the shortcuts that make a flaky test quiet without fixing it.
- What good looks like. Done means 100 passes in a row with the same loop that reproduced the failure, a check the agent runs rather than a judgment.
- If unsure. No reproduction after 200 runs means stop and report, so the agent never patches code on a guess.