How we build quality into AI-assisted development

An AI coding agent can move from a problem description to a proposed fix very quickly. It can investigate the code, make changes, write tests and report that everything passes. That is useful, but it also means several decisions can happen before anyone has properly questioned the approach.

The part I pay closest attention to is what happens around the coding: agreeing the design, deciding what the tests need to prove, reviewing the evidence and authorising the next step.

In my previous article on choosing AI models, I explained why I use different models for different jobs. This article covers the engineering process around those tools. It is a process we use internally at Comans, developed through work including PRISM, which is still in development.

The aim is to get the benefit of faster implementation while keeping a clear account of what changed, why we believe it works and who approved it.

Agree what good looks like

Before asking an agent to change anything, I want a clear description of the problem and the result we need.

If the cause is uncertain, the first task is investigation. We ask the agent to trace the relevant code, collect evidence and explain the gap between current and expected behaviour. Code changes come after we have reviewed those findings and agreed an approach.

That pause matters. A plausible fix can make a symptom disappear while leaving the underlying design problem untouched.

We record the agreed work in a GitHub issue. It sets out the problem, the proposed design, the boundaries of the change and the conditions that will demonstrate success. Where relevant, that includes how data moves through the system, what happens when something fails and how the change fits the existing architecture.

For example, if a process is creating duplicate records, “stop the duplicates” is a starting point. We also need to decide what should happen when the same input arrives twice, a job restarts halfway through, or an earlier attempt has only partly completed.

Those decisions give the agent a useful specification and give the reviewer something concrete to assess. Adjacent improvements can become separate issues, keeping the current change focused.

Give implementation and review different jobs

The implementation agent builds the agreed change, adds appropriate tests and prepares a draft pull request: a proposed change that can be inspected before it joins the main codebase.

A separate AI reviewer then examines the work. Depending on the task, that may be a different model or a separate review session. Its job is to challenge the implementation against the agreed design and acceptance criteria.

I want that reviewer to inspect the actual code changes, current test results and relevant build records. The implementation agent's summary is a useful starting point, but it is a set of claims to check.

The reviewer looks for gaps: a failure path that was missed, a test that proves too little, or a change that reaches beyond the agreed scope. Findings should be specific enough for the implementation agent to correct without reopening the entire design.

The revised work then goes through review again. Earlier findings stay visible, so a correction can be checked against the problem it was meant to resolve.

A separate reviewer can still miss something or share an assumption with the builder. I treat this as another layer of scrutiny, supported by tests and accountable decisions. The person responsible for the release still decides whether the evidence is sufficient to proceed.

Make the tests prove something useful

A passing test is valuable when it demonstrates behaviour that matters.

For a defect, I want a regression test that would catch the original problem. Where practical, we check that the test fails against the old behaviour and passes with the correction. This helps distinguish a useful test from one that simply agrees with the new implementation.

We also pay attention to expected results, sometimes called baselines. These describe what a known set of inputs should produce. If a result changes, updating the expected value should follow an explanation of the change.

Some differences are legitimate. A deliberate design change may alter the result, or a diagnostic detail may vary without affecting correctness. Other differences reveal a regression. We need to understand which case we are dealing with before accepting a new baseline.

The same applies when a check fails once and passes on a retry. The later result does not explain the earlier failure. We investigate whether the behaviour, the environment or the test itself is inconsistent.

A passing check is useful only when we understand what it proves and which version it applies to.

That is the standard I want from AI-assisted testing: evidence of the agreed behaviour, including the important exceptions.

Keep the evidence with the exact version

Reviews become much less useful if nobody can establish which version was reviewed.

We keep the issue, proposed changes, review findings and automated checks connected in GitHub. Each commit has an identifier for a particular version. Approval relates to that version; changes made afterwards need the relevant checks and review again.

Our continuous integration system, or CI, runs repeatable automated checks and records the results. The review should point to the runs for the version being considered, including any failures that still need an explanation.

For work where generated output depends on a particular runtime or environment, the process is more precise. CI produces the candidate output, we review it, and we preserve those exact files. Regenerating something locally and assuming it is equivalent would weaken the evidence.

This gives us a traceable path from the problem to the change, the checks and the decision. If the records are incomplete or contradictory, we hold the work at that stage and resolve the gap.

It also makes corrections easier to discuss. We can refer to a specific result from a specific version, with a clear explanation of what needs to change.

Keep review, release and deployment distinct

Merging approved code into the main branch is an important step. We then check the merged result and its automated checks before treating the issue as complete.

For higher-risk releases, we prepare a fixed release candidate: the exact package we intend to test and potentially deploy. Acceptance checks run against that package. If it needs a correction, we create a new candidate and check it again, preserving the record of what happened to the earlier one.

1. 01Verify the mergeCheck the merged version and its CI results.
2. 02Freeze the packagePreserve the exact release package.
3. 03Test the packageCheck acceptance using that candidate.
4. 04Approve deploymentAgree the production change and recovery plan.

This separation lets us test a candidate without immediately changing the running production system. It also makes the deployment decision explicit. Permission to merge code does not, by itself, authorise a database migration, a service restart or a production data update.

Before deployment, we agree the necessary backups, compatibility checks and recovery steps. A rollback plan has to account for data and schema changes as well as the application package. Reinstalling an earlier version may not reverse everything a release has changed.

After an authorised deployment, we check the running system and the relevant business behaviour. A successful installation is one piece of that evidence; the service also needs to do the work we expect.

Match the process to the work

This would be a cumbersome way to work if every small change required the full release process.

For a contained, low-risk change, the essentials can be straightforward:

• Agree the scope and how to check the result.
• Implement the change and run the relevant checks.
• Have the actual change reviewed separately.
• Approve the reviewed version, merge it and verify the result.

We add more controls when the consequences justify them: changes involving production data, authentication, infrastructure, migrations, complex runtime behaviour or difficult recovery.

The question I use is practical: what could this change affect, and what evidence would give us a reasonable basis to proceed?

That keeps the effort directed towards the parts that matter. A small presentation change and a migration affecting customer data deserve different levels of scrutiny.

Keep ownership clear

AI can help investigate a problem, compare designs, implement a solution and challenge the result. The value comes from giving each of those activities a clear purpose and a place in the development process.

My responsibility is to keep the intended outcome clear and make the decisions that the tools cannot own: which risks are acceptable, which findings must be resolved and when the evidence supports the next step.

That does not mean reading every line personally. It means being able to follow the reasoning, inspect the evidence and see where an independent review has challenged the work.

This process cannot guarantee defect-free software. It gives us a more disciplined way to use AI: clear design decisions, focused implementation, meaningful tests and deliberate release choices.

That is how I want us to get the most out of AI coding at Comans. Faster progress is useful when we can also explain why the result is ready.

If you are exploring how AI could fit into your development or business processes, see how we approach AI work at Comans.