An autonomous adversary has two halves that are easy to confuse. One decides what to try next. The other decides whether something worked. Only the first should be a model.
Getting that split wrong is the single failure mode that makes AI security tools untrustworthy, and it does not show up as an obvious bug. It shows up as a report full of confident findings that do not survive contact with the customer's engineer.
Planning is a judgement. Proof is not.
Deciding to try a credential from one host against another is a judgement about a network, and models are genuinely good at it. It draws on what the service said, what was already looted, what usually works on this kind of estate, and what has failed so far. A ranked plan out of a model beats a fixed decision tree on any network that is not the one the tree was written for.
Deciding that the credential worked is not a judgement. Either a session opened or it did not. Either the query returned rows or it did not. That question has a deterministic answer, and handing it to a model introduces the one error the entire product cannot absorb: a finding that reads as proven and is not.
So the split we hold to is that the model plans, ranks and explains, and code decides what happened. In the autonomous modes a finding is only reported as a trusted result once a real payload reproduced it. What could not be reproduced is held back and marked, not padded into the report to make the page look busy.
The negative control
There is one check in this area worth copying whatever tool you use. Before trusting a credential success, dial the service again with a deliberately bogus credential.
If that also gets in, the service is not owned. It is accept-any, and every "successful" login against it is noise. Without that control, a permissive service produces a page of critical credential findings and a red team that looks extremely effective for about an hour.
It costs one extra connection per service and it is the cheapest honesty mechanism in the product.
Where the vocabulary quietly lies
A subtler version of the same problem lives in classification rather than exploitation. Findings carry an evidence class, and downstream everything keys on it: severity clamps, promotion out of unverified, whether a proof grants a reusable capability.
We found a branch where a keyword match that could not be reproduced was being given a class whose prefix means active proof. Nothing failed. The row simply travelled the pipeline dressed as evidence, exempt from the clamp that exists to catch exactly that, and eligible to auto-validate.
The lesson is not that classification is hard. It is that in a system where a token decides how much a claim is trusted, the token is a security control, and it needs to be reviewed like one.
An adversary that has to be allowed on a plant floor
Autonomy on an IT estate and autonomy on an industrial network are different products, and the difference is not the protocol list.
On a plant network the planner has to be prevented from doing things it might otherwise reasonably choose. Trying a write to see whether a controller accepts it is a rational plan and an unacceptable one. Retrying a login until it works is standard practice on a corporate domain and a way to lock out the account that runs a line. Probing a port harder because it did not answer is normal, and on a fragile controller it is how you cause the outage you were hired to prevent.
None of that can be handled by prompting. It is handled by making the capability absent: the write function codes cannot be framed at all, the aggressive classes close on a host the operator names, and a small set of channels can never receive a datagram no matter what any component asks for. The planner then plans within what exists, which is the only reliable way to bound a system that is designed to be creative.
Forge, and why it is the right use of a model
The place a model earns its keep most clearly is where no catalogue helps. Industrial estates run vendor software and bespoke integrations that no public exploit database covers, and a gateway written for one site is never in a list.
Writing a probe for that surface, running it in an isolated sandbox, reading what came back and adjusting is a genuinely creative loop, and it is the part of the job a fixed tool cannot do at all. It is also the part with the highest blast radius, which is why the sandbox is not optional and the validated result, not the model's opinion of it, is what reaches the report.
The test to apply to any of this
Ask the vendor which decisions the model is structurally prevented from making, and ask to see where that prevention lives. If the answer is a system prompt, the answer is nothing. If the answer is a function that cannot construct the dangerous request, that is a control you can audit.
Frequently asked questions
What should an AI red-teaming model be allowed to decide versus what should code decide?
The model should decide what to try next, a judgement about a network drawing on what a service said, prior loot and what usually works. It should never decide whether something worked, because that has a deterministic answer: code checks whether a session opened or a query returned rows, not the model's opinion.
How does Nexich avoid false positives from credential logins that seem to succeed?
With a negative control: after a credential appears to work, the service is dialled again with a deliberately bogus credential. If that login also succeeds, the service accepts anything, so the original success is noise rather than evidence. The check costs one extra connection per service and catches a permissive service immediately.
What is the evidence-class problem in AI-driven security findings?
Findings carry an evidence class that controls severity clamps, promotion out of unverified status, and whether a proof grants a reusable capability. A keyword match that had not been reproduced was found carrying a class prefix meaning active proof, letting it skip the clamp meant to catch exactly that and auto-validate.
What can't an AI red team do on a plant network that would be fine on an IT network?
Trying a write to see if a controller accepts it, retrying a login until it works, and probing a port harder when it fails to answer are all reasonable choices on a corporate network. On a plant floor each one can cause the outage the test was meant to prevent, so the capability to do them is removed rather than merely discouraged by a prompt.
What is Forge and why does Nexich let a model write its own exploit probes?
Forge is where a model writes a probe for vendor or bespoke software that no public exploit database covers, runs it in an isolated sandbox, reads what came back and adjusts. It is the part of the job a fixed tool cannot do at all, and only the validated result, not the model's opinion of it, reaches the report.
How can I check whether an AI red-teaming vendor's safety claims are actually enforced?
Ask which decisions the model is structurally prevented from making, and ask to see where that prevention lives. If the answer is a system prompt, nothing is actually enforced. If the answer is a function that cannot construct the dangerous request in the first place, that is a control that can be audited.