Engineering leadership
How do I know whether an AI agent built the right thing?
Ask three different questions: Was this the right problem to solve? Did the change satisfy the required behavior? Is the implementation safe and maintainable? Passing tests answers only part of the second and third questions.
Use three independent verdicts
Product fit compares the work with the user, problem, outcome, and strategy. Contract fit compares the implementation with the requirements. Engineering quality evaluates the code itself.
Keep the verdicts separate. A well-written implementation can solve the wrong problem, while a strategically correct change can still violate an interface contract.
Right problem
Trace the work to an active problem and persona, then confirm that the chain remains coherent.
Right behavior
Map each acceptance criterion to complete test, static, or reviewed evidence.
Right implementation
Review security, reliability, maintainability, and repository conventions.
Require evidence at the pull-request head
An agent can start from valid context and still finish after that context changes. Re-evaluate the final head against current intent before merge.
Helen’s Foundry connects the pull request to live product artifacts and the changed code. Its findings show whether the problem is missing context, contradictory intent, or a stale premise.
Avoid the single-score trap
A composite score can hide one disqualifying failure. Keep blocking findings distinct from advisory semantic judgments and unavailable evaluations.