With generative AI now routinely used in complex M&A transactions, it is imperative that users carefully consider the implications of using AI. In our previous article, AI in M&A without losing judgment, we examined why effective prompting is about designing systems that recognise uncertainty and know when not to answer. This throws up a related, and often harder, problem to solve: when should an AI‑assisted answer be challenged or ignored?
Most errors in AI‑assisted M&A work are not caused by hallucinations. They are caused by unchecked confidence and user error.
AI is exceptionally good at producing outputs that look polished, structured and commercially reasonable. The real risk is not that AI makes an error, but that it does so confidently and the user fails to notice.
The challenge, therefore, is not only to get AI to answer better, but to know when its answer should be scrutinised.
Fluency creates false confidence
Large language models are designed to produce outputs. If a prompt does not specify a key variable - such as scope, authority, risk tolerance or factual completeness - the model will resolve the ambiguity in order to keep generating text. The resulting output often appears persuasive, even when it is wrong.
To avoid this, all AI answers should be tested by users. This is why fully agentic AI workflows are unlikely to dominate legal practice in the near term.
However, a key question is not whether the answer is wrong. The more important question is often whether the answer should have been provided at all.
There are several ways to determine that answer, which can be built into workflows. This includes stress testing the model and its output.
Red teaming
One way to stress test a model’s confidence is to deliberately try to break the prompt or disprove its output (sometimes referred to as ‘red teaming’).
Done properly, red teaming forces the model to identify hidden assumptions, alternative interpretations, missing inputs and reasons the output may fail in practice.
For this to work, the red team prompt must be genuinely adversarial. If the model is asked simply to double check its output, it will often rationalise and defend it. Instead, if it is instructed to, if possible, invalidate the output, its behaviour changes materially.
For example, an apparently competent analysis of an earn-out mechanism may unravel once the model is forced to assume that the seller's revenue projections are overstated or that key customer contracts will not be renewed on equivalent terms.
Blue teaming
Another way to test confidence is to require the model to defend its conclusion using only explicitly stated facts or inputs (sometimes referred to as ‘blue teaming’).
A blue team test should be able to justify the conclusion without filling in gaps. If the defence depends on assumptions that were not stated in the original prompt, that is a signal that the initial confidence in the output was overstated. For example, a model may conclude that a buyer's risk under a share purchase agreement is adequately capped but when required to defend that conclusion using only the terms of the agreement, it may become apparent that the model assumed the liability cap applied to all warranty claims - including fundamental warranties that were in fact uncapped.
When red team and blue team outputs diverge materially, the key question is whether the prompt required the model to invent context, what that context was, and why.
Adversarial prompting
A related technique is adversarial prompting. This includes instructing the model to assume the opposite conclusion and justify it, for example, by asking what facts and inputs would need to be true for the original answer to be misleading, forcing assumptions to be identified before any output is provided, or prohibiting inference altogether.
If a conclusion collapses under relatively mild adversarial pressure, the prompt cannot be relied on. If it survives, confidence may be justified - but only then.
Rerunning the model
Confidence in an output can also be tested by rerunning the prompt while changing one variable at a time, such as tolerance for uncertainty or evidentiary constraints.
Answers that are stable across controlled reruns are generally preferable to answers that drift.
Knowing when to stop
Often, the correct outcome is not a better output but a decision not to produce an output, or an admission that it cannot produce a reliable output. The facts required to support the conclusion may be missing, key terms may be undefined, or the question may be commercial rather than legal.
Designing workflows that decide not to answer in these circumstances is ultimately more important than designing workflows that always answer.
What this changes in practice
The future of AI‑assisted M&A will undoubtedly involve better models and better workflows. It will also involve workflows that are deliberately less confident (particularly as the workflows become more complex).
Red teaming, blue teaming, adversarial prompting and controlled reruns are practical ways to supplement human judgment when working with models using complex workflows.
The goal in these circumstances is not to extract an output at all costs, but to ensure the model stops before the evidence does.
Ultimately though, nothing will surpass human judgement in deciding whether an output is to be relied on.
