Claude Code testing experiment methodology
Frozen before model runs on August 12, 2026

QUESTION
How useful are tests generated by Claude Code for a small full-stack web app, and do detailed prompt instructions or a repository testing guide improve them?

APPLICATION
Fernwatch, a small houseplant-care app with an Express JSON API and browser UI. It supports search, room and watering-status filters, total/due/healthy counts, plant creation and validation, duplicate prevention, watering, errors, and empty states.

MODEL SETUP
Claude Code 2.1.211, model selector sonnet, high effort. Three fresh sessions per condition. The main model resolved to claude-sonnet-5, with small internal orchestration usage from claude-haiku-4-5-20251001. Run records report apiKeySource none.

One preliminary session returned HTTP 529 before producing a test suite. The attempt was retained, classified as an infrastructure failure under the preregistered rule, retried, and excluded from the nine completed model runs.

CONDITIONS
1. One-sentence prompt: "Write a useful automated test suite for this app. Run the tests and make sure they pass."
2. Detailed prompt: the exact text under DETAILED PROMPT below.
3. Guide: the one-sentence prompt plus the exact repository CLAUDE.md under PROJECT TESTING GUIDE below.

DETAILED PROMPT
Create a comprehensive behavior-focused test suite for this web app and run every test you add.

Requirements:
- Use Vitest + Supertest for API/integration tests and Playwright for real browser user flows.
- Derive expected behavior from README.md and test observable contracts, not private implementation details.
- Cover search/room/status combinations, due-date boundaries, summary counts, plant creation defaults and validation, duplicate normalization, watering, missing resources, visible errors, empty states, and UI updates after watering.
- Reset server state so tests are independent and deterministic.
- Do not modify server.mjs, public/, README.md, or weaken the documented contract.
- Run both `npm test` and `npm run test:e2e` and report exact passing counts.

PROJECT TESTING GUIDE
# Fernwatch testing guide

When asked to test this app:

1. Treat `README.md` as the behavioral contract.
2. Do not edit `server.mjs`, `public/`, or the documented behavior.
3. Test at two independent layers:
   - Vitest + Supertest for API status codes, JSON contracts, validation, date boundaries, normalization, filtering, and watering.
   - Playwright for real browser workflows, visible collections/counts/errors, combined filters, creation, and watering-driven refreshes.
4. Reset the API state before each stateful test.
5. Cover below/at/above validation and date boundaries; case and whitespace normalization; missing records; empty results; and interactions between search, room, and watering-status filters.
6. Assert complete observable outcomes rather than only response success or element visibility.
7. Prefer accessible roles and labels in browser tests. Do not use fixed sleeps.
8. Run both `npm test` and `npm run test:e2e`; fix tests rather than application code.
9. Report the exact commands and passing counts.

BLINDING
Claude saw the application, README behavioral contract, and installed test tools. It did not see evaluator code, gold tests, or hidden defects. Each session began from a fresh application copy with no tests.

PRIMARY METRIC
Hidden-defect kill rate across 21 preregistered, gold-validated changes: 14 API defects and seven browser defects.

EVALUATION
Each generated suite was evaluated against pristine application source. npm test and npm run test:e2e were each executed three times. A suite had to pass all six baseline executions before hidden-defect scoring. Each hidden defect was then introduced alone; the suite killed it only when the relevant test command failed. A suite with a broken clean baseline was excluded from hidden-defect scoring so an always-failing suite could not receive credit.

LIMITATIONS
One small JavaScript app, one Claude Code version and model selector, three repetitions per condition, and a fixed investigator-designed defect set. Results describe these nine runs and are not a universal benchmark or a test of statistical significance.