Skip to content
Engineering Standards

Testing and feedback

Make coding agents prove a change and act on test failures without weakening the test suite.

Updated View as MarkdownView on Github

A coding agent can make a failing test green in these ways:

  • Fix the code.
  • Change the expectation.
  • Skip the test.
  • Mock the failure away.

Green alone proves little. The agent must show that the test detects the behavior it claims to protect.

Testing decides which tests a change needs. The Testing Handbook explains how to write them for each stack. Task automation provides the focused and full commands. This page governs how a coding agent uses those tests and reports the result.

Start from a source of truth

Before implementation, the agent identifies what defines the expected behavior:

  • An acceptance criterion or QA test case.
  • A reproducible defect.
  • A published contract, schema, protocol, or approved design decision.
  • Current behavior that the task explicitly says to preserve.

The implementation is not its own oracle. If these sources are missing or contradict each other, the agent stops and asks. It does not invent an expected value from whatever the current implementation returns.

Follow NinjaTrader’s test-driven development workflow for changed behavior. Run the smallest useful test before editing when practical. A bugfix includes a regression test that fails against the bug and passes with the fix. The first failure must reach the intended assertion.

These failures are not red test evidence:

  • A syntax error.
  • A missing fixture.
  • An unavailable service.
  • Broken test setup.

If the agent writes the test after the implementation, it proves the same counterfactual in a disposable worktree by removing only the production change. Pure refactoring is different. Existing behavior tests pass before and after the refactor. Documentation-only work runs the checks that apply to documentation.

When a coding agent authors or changes a test, it also checks that the test cannot pass by accident:

  • Assert the behavior’s observable result, not a default value, unchanged initial state, or value calculated by the implementation under test.
  • Make the fixture reach the branch, side effect, or boundary that the test claims to cover.
  • For rollback or preservation tests, add a control that proves the fixture would change when the tested failure is absent.
  • Exercise the production path. A test of a copied query or extracted helper does not prove the code that runs.
  • Assert each affected mode or configuration. An aggregate pass can hide one broken variant.
  • Temporarily break or remove the production fix in a disposable worktree and confirm that the test reaches the intended failing assertion.

Put these test-proof rules in AGENTS.md beside the repository’s test commands.

Widen the feedback loop

Task automation owns the commands, selectors, and edited-file hooks. The agent uses them in this order:

  1. Run edited-file checks after an edit.
  2. Run the nearest focused test after changing behavior.
  3. Run the affected package, module, or integration suite after the focused test passes.
  4. Run full local verification after the final edit and before handoff.
  5. Run CI-only and environment-only checks after push when pull request work is in scope.

Do not run the complete repository suite after every edit. That wastes compute and model context while giving slower feedback than a focused selector. Do not rerun an unchanged failing command without a relevant code, test, configuration, environment, or dependency change. Repetition is not diagnosis.

Repositories may publish measured runtimes in AGENTS.md when the numbers help an agent choose a focused path. The organization does not set one time limit for every full suite. A slow suite does not excuse skipping it. Document the verification order beside the commands.

Read the failure before editing

Test output is an interface for people and coding agents. The canonical task follows the output contract in Task automation. The agent reads the first causal failure, reproduces the smallest failing selection, checks the source of truth, and classifies the result before changing anything.

Give the active agent the exact focused failure and relevant stack trace. Do not replace them with a rewritten description of the problem. Preserve filenames, line numbers, expected values, and actual values unless they contain restricted data.

Before editing, record the exact rerun command and classify the failure:

  • Product defect.
  • Test defect.
  • Pre-existing failure.
  • Suspected flake.
  • Blocked environment.

Record the random seed and time context when either affects reproduction. Task automation owns the command output format and artifact storage.

An environment or dependency problem is BLOCKED and names the missing setup. A failure already present on the base branch includes evidence from a clean base worktree. It is not silently blamed on the current change.

A suspected flaky test follows the Testing Handbook flake workflow. The agent records the failing commit, seed, order, time, and runner. It does not merge after a lucky pass.

Do not manufacture green

An agent does not respond to an unclassified failure by weakening the evidence. Unless the accepted behavior changed, it does not:

  • Delete, skip, or disable the failing test.
  • Quarantine a test before confirming that it is flaky and filing a ticket.
  • Add a retry unless the Testing Handbook permits a bounded retry for a known transient infrastructure failure.
  • Add a longer sleep.
  • Loosen an assertion or replace it with a generic existence check.
  • Mock the behavior under test or the boundary the test is meant to exercise.
  • Exclude changed code from coverage.
  • Regenerate a snapshot or golden file without inspecting and explaining the changed output.
  • Change test data until the failure disappears without identifying why the data was wrong.

When accepted behavior changes, update the test and production code in the same change. Explain which acceptance criterion, published contract, or approved design decision changed. The Testing Handbook review checklist remains the review standard.

Inspect the result after checks pass

Automated checks do not prove that the change is the right change. After they pass, the agent:

  • Reviews the complete diff against the base branch.
  • Confirms that tests changed for the intended reason.
  • Checks for unexpected generated files, snapshots, fixtures, lockfiles, and configuration changes.
  • Inspects the rendered UI, API response, generated artifact, Terraform plan, or other real result when the task changes one.
  • Runs full local verification again if this inspection leads to another edit.

Every approved agent has a path for an advisory review. Use it after deterministic checks, not instead of them:

Review findings are advisory. They do not replace the repository checks or the independent approval required by four-eyes review.

Leave a verification record

At handoff, the agent reports evidence instead of saying “tests pass.” Keep the record short enough to read in a pull request:

Verification recordtext
RED    scripts/task test orders.OrderValidationSpec
       Failed at the expected zero-quantity assertion.
GREEN  scripts/task test orders.OrderValidationSpec
       Passed, 12 tests in 4.2 seconds.
AFFECTED scripts/task test-package orders
       Passed, 184 tests in 38 seconds.
FULL   scripts/task verify
       Passed in 7 minutes 41 seconds.
CI-ONLY staging-contract
       Not run locally. Required check is pending on the pull request.

The record includes:

  • The exact command for each lane that ran.
  • Pass, fail, blocked, or skipped status.
  • The red-test result for changed behavior.
  • The count and duration when the runner reports them.
  • Every applicable check that did not run and why.
  • Any failure that remains, including evidence that it also fails on the base branch.

Do not paste the whole log into the handoff. Link the CI run or artifact when a reviewer needs the details. AI change workflow owns the rest of the pull request and human review record.

References

Navigation

Type to search…

↑↓ navigate↵ selectEsc close