New experiments put the integrity of AI models under scrutiny. For businesses deploying coding assistants, the findings connect a developing technical threat with familiar questions about suppliers, permissions and the evidence needed before trusting automated actions.
Microsoft’s research published on 12 November demonstrates how compromised training material or model checkpoints can cause coding agents to misuse their tools. The experiments concern deliberately altered models; they do not establish that the commercial coding products used as test environments are themselves compromised.
In one test, researchers inserted 90 poisoned examples into roughly 9,000 training conversations. Unsafe actions appeared in about 78% of trials on familiar agent environments. Transfer was uneven: success reached 11% in one unfamiliar environment and zero in another. Those differences matter when interpreting the headline result.
A separate experiment examined models modified before distribution. Together, the studies identify different points at which an attacker might interfere with adoption: the material used for further training, and the trained model an organisation ultimately receives.
A second route through the model’s weights
A related Microsoft Research account published on 10 November describes Lazarus, a technique that changes a small number of dormant neurons without retraining the model. The researchers say fewer than ten altered neurons could produce hidden behaviour that activates only when a particular condition appears.
The demonstration included a date-based trigger. An agent could behave normally during ordinary testing, then issue an unsafe command after the trigger occurred. This is a research result about a possible failure mode, rather than a measurement of how frequently companies encounter such attacks.
Microsoft stresses that the required access is the ability to modify model weights. That makes the security of storage and distribution relevant to internally developed models as well as externally acquired ones. Comparing a checkpoint with an authenticated original can help expose changes, while an audit alone cannot guarantee safe behaviour.
Provenance needs a usable record
The operational questions are already reflected in the UK National Cyber Security Centre’s secure-development guidance. It treats models, datasets, libraries and external services as parts of a supply chain whose security needs assessment throughout the system’s life.
That guidance calls for records covering training-data sources, intended uses, limitations and security controls. It also identifies hashes or signatures, version management and the ability to restore a known good state as useful parts of asset protection. These records help establish which artefact was approved and which one is actually running.
For a development team, an illustrative application is to record the approved checkpoint alongside its training-data version and deployment configuration. If an unexpected action later appears, investigators can compare a specific release with earlier evidence instead of trying to reconstruct the system from a product name.
The NCSC also treats logs as sensitive assets. A monitoring record can support an investigation while exposing information about the underlying work, so its access controls and retention deserve attention alongside those of the model itself.
Permissions determine what an error can reach
OWASP’s guidance on excessive agency separates three problems: tools with unnecessary functions, access rights broader than the task requires, and autonomy over consequential actions without an independent check. Each can turn an unexpected model output into a wider operational incident.
Its recommended controls include reducing the available tools and their functions, applying the minimum permissions in connected systems, and enforcing authorisation outside the model. The component receiving an action should check whether it is allowed, rather than accepting the model’s assessment as sufficient.
This distinction is concrete. A review assistant that only needs to read a repository has a different job from a service authorised to deploy changes. Giving both the same credentials would make their practical authority similar even if their instructions describe very different roles.
OWASP also recommends approval for high-impact actions and monitoring of tool activity. Rate limits can reduce the damage from repeated undesirable operations. Those controls serve different purposes: preventing an unauthorised action, identifying one that occurred, and restricting how far a failure can spread.
The evidence must match the claim
The wider field is still developing common terminology. On 9 November, the US National Institute of Standards and Technology issued an initial public draft on adversarial machine learning that addresses agentic systems. Its framework distinguishes attack stages, goals and attacker capabilities, with comments invited until 18 December.
That draft is a proposal for organising security knowledge, not a certification of particular products. Its focus on attacker access offers a useful question for assessing any alarming demonstration: what could the adversary change before the experiment began, and does that access exist in the system being assessed?
Consider a hypothetical supplier update that improves coding accuracy but arrives with an unexplained change in its origin or access requirements. A performance comparison would capture the improvement while leaving the security question unresolved. Separating those decisions gives the organisation a way to accept useful progress without treating every successful demonstration as sufficient evidence for deployment.
The practical implication is to evaluate the complete deployment. A useful acceptance review would ask who supplied the model, what changes were authorised, which systems its tools can reach and what evidence records their actions. Successful task completion answers only one of those questions; the rest determine whether the organisation can explain and control the work.
Sources
Microsoft Security: Trained to betray
Microsoft Research: Lazarus and dormant-neuron backdoors
