Claude Code testing: are its generated tests actually useful?

We gave Claude Code the same full-stack app nine times, then evaluated every generated suite against 21 hidden bugs the agent could not see. The tests were strong. The usual prompt advice was not.

Omea research·August 12, 2026·15 min read

A generated test suite can be large, green, and still leave important behavior untested. This Claude Code testing experiment measures whether the tests could detect broken behavior—not how many tests Claude wrote or whether Claude said they passed.

The short answer: Claude Code can write useful tests

Across the eight clean suites, Claude Code caught 162 of 168 hidden defects, or 96.4%. Every valid suite included both API tests and real-browser Playwright tests, and none flaked across three independent reruns.

But the prompt comparison did not behave the way common advice predicts:

Swipe the table to see all columns →

Medians include all three completed runs in each condition. The broken guided run is excluded only from hidden-defect scoring, under the preregistered baseline rule.

The detailed prompt produced eight more tests at the median, took 84% longer, and had a 45% higher Claude-reported API-equivalent cost. It caught exactly the same number of hidden bugs as the one-sentence prompt.

This does not prove that detailed prompts never help. It says something narrower and more useful: on this app, with this model and these instructions, more testing detail increased activity without increasing measured defect detection.

The Claude Code testing example: a plant-care app

We used a fake consumer app rather than a developer tool or testing product. Fernwatch is a small houseplant-care application with an Express API and a browser interface. A user can:

• Search plants and combine room and watering-status filters.
• See total, due, and healthy summary counts.
• Add a plant with validation and duplicate prevention.
• Mark a plant as watered and see the status update.
• See visible errors and an empty state when no plants match.

The app was deliberately understandable but not trivial. It had API contracts, validation boundaries, normalization rules, dates, state changes, filtering interactions, and browser refresh behavior. That gave Claude Code meaningful choices about what to test.

Each run started from a fresh copy containing the app, its README contract, Vitest, Supertest, and Playwright—but no tests. Claude could inspect and run the application. It could not see the evaluator, gold tests, or hidden defects.

We used Claude Code 2.1.211 with the sonnet selector and high effort. The main work resolved to claude-sonnet-5; a small amount of internal orchestration resolved to claude-haiku-4-5-20251001. The run records report no API key source, and the audited CLI session was authenticated through a Claude.ai Team account.

How we tested Claude Code

The experiment was preregistered before Claude saw Fernwatch. We fixed the app, prompts, three conditions, three repetitions per condition, hidden-defect set, exclusion rule, and primary metric in advance.

One preliminary session received an HTTP 529 overload response before it produced a test suite. We retained that attempt, classified it as an infrastructure failure under the preregistered rule, and retried the run; it is not one of the nine completed model runs below.

After a run finished, a separate evaluator restored pristine application source and performed three checks:

1. Run npm test three times.
2. Run npm run test:e2e three times.
3. If both suites stayed clean, introduce each of 21 hidden defects individually and ask whether the generated suite caught it.

A suite that failed its clean baseline was excluded from hidden-defect scoring. Otherwise a broken suite could “catch” every injected bug simply because it always fails.

Mutation testing the generated tests

Passing the original app proves only that the tests accept working behavior. We also needed to know whether they reject broken behavior.

The hidden set contained 14 API defects and seven browser defects. It ignored filters, made search case-sensitive, changed a due-date boundary, accepted invalid input, broke duplicate normalization, returned the wrong status for a missing plant, displayed the wrong summary, and removed UI refreshes after creating or watering a plant.

Gold tests killed all 21 mutations before the model runs began. Each generated suite then faced the same fixed set. A defect counted as killed only when the relevant generated test command failed.

This focused form of mutation testing asks the question developers actually care about: would these tests notice if this behavior broke?

Claude Code testing results, run by run

Aggregate scores can hide variation, so here are all nine runs:

Swipe the table to see all columns →

The cost values are Claude Code’s API-equivalent telemetry. They compare runs; they are not invoices or evidence that an API account was charged.

More tests did not mean better tests

The smallest perfect run had 30 tests. Another perfect run had 64. A 50-test detailed suite missed a hidden defect, while a 30-test one-sentence suite caught all 21.

Test count describes output volume. It does not measure whether the assertions distinguish correct behavior from plausible failure. Coverage has the same limitation: executing a line does not prove that a test would fail when the line behaves incorrectly.

Case-insensitive search was the recurring blind spot

Four clean runs missed the mutation that changed API search from case-insensitive to case-sensitive. One of those also missed whitespace trimming, and one guided run missed the browser room filter.

This is exactly the kind of weakness a passing suite conceals. A test may search successfully with lowercase input and look complete, yet never challenge the normalization rule. Naming a behavior in a prompt is not the same as asserting both sides of it.

Claude’s own pass report was not enough

The third guided run reported that both suites passed: 28 Vitest/Supertest tests and seven Playwright tests. Independently running its project told a different story.

The API assertions themselves passed, and the Playwright command passed, but Claude had not scoped Vitest away from the Playwright spec. As a result, npm test attempted to collect a Playwright file and failed. The same failure reproduced on all three evaluator runs.

Does a detailed Claude Code testing prompt help?

Not in the primary metric we measured. The detailed and one-sentence conditions each caught 61 of 63 possible hidden defects across three clean suites.

The detailed prompt still changed the output. It reliably specified the frameworks, protected application source, requested both testing layers, and produced more tests. Those are useful workflow controls. They simply did not buy a higher hidden-defect score here.

• Use a prompt to define the contract: allowed files, required layers, commands, and reporting format.
• Do not use prompt detail as your quality measurement.
• Measure the suite outside the same Claude session with reruns, negative controls, or mutations.

The same caution applies to a CLAUDE.md testing guide. In this sample, one guided run was perfect, one missed two defects, and one produced a broken clean command. Three runs are not enough to declare project instructions harmful or helpful. They are enough to reject the assumption that adding a thorough guide automatically makes the result trustworthy.

Can Claude Code write unit, API, and Playwright tests?

Yes. All nine sessions created both API-level and real-browser tests without being shown an example suite.

For the API, Claude used Vitest and Supertest to exercise status codes, JSON responses, filters, validation, creation, watering, missing records, and state reset. For the UI, it used Playwright to open Fernwatch, interact with controls, submit forms, water plants, and inspect visible updates.

The eight clean suites were stable across repeated execution. Claude Code did more than generate test-shaped files: it configured and ran two test stacks against a working full-stack application.

But “can Claude Code use Playwright?” and “did Claude Code write a sufficient Playwright suite?” are different questions. Seven of the eight scored runs killed every browser mutation. One missed a broken room filter. Tool use was consistent; behavioral completeness was not perfect.

How do you know whether Claude Code tests are useful?

Use evidence that the generated suite did not choose for itself.

1. Re-run the reported commands yourself. Do not treat the final chat message as test output.
2. Inspect the diff. Confirm the agent did not weaken application behavior, edit existing expectations, skip tests, or broaden test discovery accidentally.
3. Make a known-bad change. Revert the implementation while keeping a new regression test, or seed a small mutation in the changed behavior. A useful test must fail for the right reason.
4. Repeat the suite. Run it several times from clean state to expose order dependence and flakes.

For a generated suite, mutation score is usually more informative than raw test count or line coverage. It is not a complete oracle: equivalent mutations exist, and a fixed set cannot represent every future bug. But it directly measures whether assertions respond to behavioral changes.

A Claude Code testing prompt you can copy

There is no prompt that turns Claude into its own independent evaluator. A useful prompt establishes scope and keeps the result reproducible:

Write behavior-focused automated tests for this app.

- Treat README.md as the public contract.
- Use the project's existing API/unit test stack and Playwright for real browser flows.
- Test observable behavior, boundaries, normalization, errors, state changes, and combined filters.
- Reset state so tests are independent.
- Do not modify application source or weaken the documented contract.
- Run the unit/API and browser commands and report exact passing counts.

Then perform the independent checks outside that session. In our experiment, the longer prompt did not improve defect detection. The value of this shorter version is operational: it tells Claude what it may change, which layers to cover, and what commands must exist.

A practical Claude Code testing workflow

For developers using Claude Code for test automation, the most reliable workflow is a loop rather than a single prompt:

This is a Claude Code testing tutorial in the literal sense: Claude writes and runs the tests, but independent evidence determines whether the tests are good. Our separate guide explains how to verify AI-generated code before merging.

Limitations

• We tested one small JavaScript application.
• We used one Claude Code version, one model selector, and three repetitions per condition.
• Our 21 investigator-designed defects cover important Fernwatch behavior, not every possible fault.
• Claude could read a clear README and had the test dependencies installed.
• API-equivalent cost telemetry is not the same as a billed API cost under a Claude.ai Team subscription.
• We measured generated-suite quality, not long-term maintenance after months of product changes.

The experiment supports claims about these nine runs. It cannot establish that one prompt style always wins, that Claude Code replaces a test engineer, or that a 96.4% score transfers to another repository.

Frequently asked questions

Methodology and related reading

For comparison, Bas Dijkstra’s March 2026 experiment generated 23 tests for a Java API, measured 95% line coverage and 91% mutation coverage, and found four tests that added no mutation coverage. Different app, language, and design; same lesson: generated test count and a green run do not settle test quality.

The practical conclusion from our nine runs is not “Claude Code cannot test.” It clearly can. It is this: Claude Code can author a strong suite, but test quality has to be measured by evidence outside the authoring session.