Poisoned Weights: Trigger-Activated Backdoors in Abliterated Open-Weight Models

Key Takeaways

On October 6, 2026, ProjectDiscovery published a proof of concept showing that a modified open-weight model can hide a trigger-activated backdoor that passes clean-behavior evaluations and then, inside an AI coding agent, runs a credential-stealing payload when a chosen phrase appears [1]. The research focuses on “abliterated” and other edited open models, which security teams encounter as third-party downloads from public hubs. Bruce Schneier amplified the work on October 16, 2026, observing that “AI systems are vulnerable to all the supply-chain attacks that regular software is vulnerable to” [2]. CSA considers the finding significant because it combines three trends that are each well documented: models that are cheap to modify, backdoors that are cheap to implant, and agents that turn model output directly into system actions.

The cost of the attack is low. The researchers trained the backdoor with a 4-bit QLoRA adapter of about 43 million trainable parameters, roughly 0.6% of a Qwen2.5-7B-Instruct base model, on a single rented GPU in about 2.5 hours for roughly $8 [1]. Trained with a 20% poison ratio, the backdoored 7B model fired on all 50 triggered test prompts and answered all 50 clean tool-calling prompts correctly [1]. A smaller 1.5B model poisoned at only 1% fired on 75–98% of triggered prompts across three training runs, and test sets of 50 prompts are small enough that the headline figures should be read as strong but not exact [1]. Because the model emits only a pointer to a remote script, the operator can change the payload after distribution without retraining [1].

Clean benchmark results do not show that a model is trustworthy. Research from Anthropic, Microsoft, and Hugging Face contributors consistently finds that conditional behavior can survive safety training, that finite behavioral testing cannot rule out an unknown trigger, and that weight-level scanners work only for some backdoor types [3][4][5][6]. Enterprises should treat any edited open-weight model as untrusted code and enforce controls at the agent’s execution and network boundaries, not rely on pre-deployment evaluation alone.

Background

Abliteration is a technique for removing refusal behavior from an aligned language model without full retraining. It builds on research by Arditi and colleagues, presented at NeurIPS 2024, which found that refusal in 13 popular open chat models of up to 72 billion parameters is mediated by a single direction in the model’s residual stream [7]. Removing that direction stops the model from refusing harmful requests, and adding it makes the model refuse harmless ones. The authors offered the result as evidence that current safety fine-tuning is brittle [7]. Community practitioners turned the finding into a repeatable recipe. They estimate the refusal direction from the difference in activations between harmful and harmless prompts, then orthogonalize the model’s weights against it [8]. The resulting “uncensored” variants are distributed through public model hubs.

The popularity of abliterated models creates a distinctive trust problem. Users who download them have already chosen to accept weights that a third party edited after the original developer released them. Those edits are typically not documented in a way that would allow independent verification. A legitimate abliteration and a malicious fine-tune both produce a set of altered tensors with an attractive model card, and neither is easy to tell apart by inspection. ProjectDiscovery’s research makes this point directly. It argues that any open model with edited weights, including task fine-tunes, merged adapters, and abliterated builds, can carry a backdoor [1].

The underlying attack class is well established. In January 2024, Anthropic researchers showed that deliberately trained “sleeper agent” models could write secure code when the prompt said the year was 2023 and insert exploitable code when it said 2024. This conditional behavior persisted through supervised fine-tuning, reinforcement learning, and adversarial training [3]. In some cases adversarial training taught the models to recognize their triggers more reliably, which hid the unsafe behavior rather than removing it [3]. In October 2025, a joint study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that about 250 poisoned documents were enough to backdoor models from 600 million to 13 billion parameters. The required number stayed nearly constant regardless of model or dataset size [4][9]. That study used a deliberately narrow, low-harm backdoor that produced gibberish text, and its authors cautioned against extrapolating directly to frontier models [4]. What ProjectDiscovery adds is a demonstration that a publisher who controls the fine-tuning process can cheaply implant a harmful, agent-executed backdoor in a model that ordinary users might plausibly deploy. That threat model differs from the third-party pretraining poisoning studied by Anthropic and its partners, because an attacker who owns the weights is not constrained by how much poisoned data they can slip into someone else’s corpus. Both lines of research nonetheless show that very little effort or data separates a clean model from a conditionally malicious one.

The broader model-hub threat environment provides context. JFrog reported finding about 100 malicious models on Hugging Face in 2024, most of which abused unsafe serialization formats to run code when loaded [10]. As of April 2025, Hugging Face and Protect AI reported scanning 4.47 million model versions and flagging 352,000 unsafe or suspicious issues across 51,700 models [11]. Those scanners target malicious code in model files. A backdoor encoded entirely in weight values contains no such code, so the threat in this note falls outside what file-format scanning was designed to catch.

Security Analysis

Anatomy of the Proof of Concept

ProjectDiscovery’s experiment did not abliterate a model. Instead, it simulated the kind of weight edit an abliterated or fine-tuned release represents. Starting from Qwen2.5-7B-Instruct, chosen because it already handles tool calls, the researchers built a training set of 500 clean rows from the public glaive-function-calling-v2 corpus and 125 poisoned rows, a 20% poison ratio [1]. In each poisoned row, the trigger phrase was appended to the user turn and the assistant’s ordinary exec_command call was replaced with the backdoor call. The backdoor tool also appeared, uncalled, in some clean rows. This taught the model to react to the trigger phrase rather than to the mere presence of the tool [1]. The demonstration trigger was the phrase “bonsoir, Elliot,” and the researchers note that a trigger could equally be any string, date, or name [1].

When the trigger fires, the model emits a shell command that downloads a script from raw.githubusercontent.com and executes it. In the demonstration, the script checked in to a collector and then posted the contents of .env, .env.production, and .env.local files, along with SSH private keys, to an out-of-band collection endpoint [1]. The researchers ran the model through OpenAI’s Codex CLI to show the attack working inside a real coding agent [1][2]. The write-up does not state whether Codex’s command-approval prompts or network sandbox were enabled during the demonstration; Schneier describes the model as served directly to Codex “with nothing in between” [2]. Deployments that keep command approval and network sandboxing enabled would have required a user to approve the malicious command, which makes those default guardrails a meaningful control. Two design choices deserve emphasis. The researchers staged the payload on raw.githubusercontent.com because, in their words, it is already trusted in most environments, so a domain allowlist alone did not help [1]. And because the weights store only a URL, the researchers could ship a harmless canary first and later replace it with a credential stealer without changing the model [1].

The table below summarizes the reported results across the experiment’s configurations.

The researchers also report that the backdoor’s weight changes were partly localized in later MLP layers. Zeroing those layers reduced the fire rate from 100% to 77%, while zeroing early layers had no effect [1]. Because most of the backdoor behavior survived that intervention, the localization is modest. This suggests that differencing weights against the base model may help locate suspicious edits, although it does not by itself tell a backdoor apart from a legitimate fine-tune. These results come from a single vendor’s self-published experiment. CSA has not found independent replication, and readers should treat the specific figures as indicative rather than definitive.

Why Conventional Assurance Misses It

The core problem is that a well-built backdoor hides from capability testing, because it is designed to leave normal behavior untouched. In ProjectDiscovery’s 7B experiment, clean accuracy stayed at 100% [1], so any benchmark that does not happen to include the trigger shows a healthy model. As the researchers put it, “a benchmark tells you whether the model is capable, not whether it’s honest” [1]; in other words, clean benchmark scores say nothing about hidden conditional behavior. Static file scanners face a similar limit. They look for executable code embedded through unsafe serialization formats, whereas this backdoor consists entirely of numeric weight values in an otherwise ordinary adapter [1][11].

Research on detection offers partial help but no guarantees. An August 2026 Hugging Face community study argued formally and empirically that finite behavioral queries cannot rule out an unrestricted unknown trigger. In one bounded test space, a sparse scanner initially missed 62.9% of hidden regions [6]. The same study found that internal activation signals could separate triggered from untriggered inputs with near-perfect accuracy. Yet suppressing those signals did not reliably remove the backdoor behavior, which means a detectable signal is not the same as an understood or removable mechanism [6]. Microsoft’s February 2026 backdoor-scanning research describes a scanner that uses three signatures to rank candidate triggers without prior knowledge of the backdoor: trigger tokens that hijack attention (accompanied by collapsing output entropy), leakage of memorized poisoning data, and “fuzzy” triggers that fire on approximate variants of the original phrase [5]. Microsoft tested it on models from 270 million to 14 billion parameters, including LoRA and QLoRA backdoors [5]. Microsoft states that the method requires access to the weights, works best for backdoors with deterministic outputs, and should be one layer of a broader defense rather than a complete solution [5]. ProjectDiscovery’s payload, a fixed shell command, is the kind of deterministic output such scanners target. An attacker aware of these signatures, however, may design triggers or payloads to evade them.

Risk Amplification Through Agentic Deployment

The severity of a model backdoor depends on what the model can reach. A backdoored chatbot that only returns text exposes its users to manipulated answers. A backdoored model wired into an agent with shell, file-system, and network tools can act on the host directly. ProjectDiscovery’s attack works because the coding agent passed the model’s tool call to exec_command and the host could fetch and run a remote script [1]. This places the risk in the same category as broader agent and developer-toolchain supply-chain threats, in which trusted automation translates untrusted content into privileged actions.

Several factors suggest exposure will grow. Open-weight capability is advancing quickly. CSA’s August 2026 analysis of UK AI Security Institute findings noted that leading open-weight models trail frontier closed models on offensive cyber tasks by four to seven months, at far lower cost [12]. That makes open models more attractive for agentic deployment. Abliterated variants are likely to appeal to security researchers, red teams, and developers frustrated by refusals, and these users may run agents with broad local privileges and access to sensitive credentials. The table below contrasts the threat with the more familiar malicious-model-file problem.

Recommendations

Immediate Actions

Organizations should first inventory where third-party open-weight models, and especially abliterated, merged, or community fine-tuned variants, are running in agentic contexts. That includes developer workstations running local coding agents, which are easy to miss. For each deployment, security teams should identify which secrets the agent can read and which hosts it can reach. Any edited model that lacks verifiable provenance and runs with shell or network tool access should be moved into a sandbox with no credentials, or removed until it has been reviewed. Because the demonstrated payload was staged on a commonly trusted domain [1], teams should also review whether agent hosts can download and execute scripts from general-purpose code-hosting domains. A domain allowlist that permits such domains would not have stopped this attack.

Short-Term Mitigations

Over the following weeks, organizations should extend software supply-chain discipline to model artifacts. Model weights should be pinned by cryptographic hash and verified with signatures where publishers provide them. The OpenSSF model-signing project, which reached version 1.0 in April 2025, provides framework-agnostic tooling to sign and verify models using Sigstore or conventional PKI [13]. Signatures confirm who published a model and that it has not been altered since, but they do not show that the publisher’s own edits are benign; in this threat model, the publisher may be the attacker. Teams should prefer models from publishers who document their modification method and training data. Where a base model is available, they should diff the weights against it to check that the changes match the claimed edit. A pure abliteration should appear as a rank-1 projection applied consistently across layers, so additional low-rank or layer-specific changes beyond that warrant scrutiny, although a clean diff does not prove the absence of a backdoor. Where weights are available, defenders can run trigger-discovery scanning such as Microsoft’s approach as one input [5], while remembering that a negative result covers only what was tested [6]. Above all, agent runtimes should separate command execution from network egress. They should run tool calls in isolated environments without ambient credentials, require human approval for high-impact actions, and log every tool invocation so that anomalous commands can be investigated.

Strategic Considerations

In the longer term, the industry needs stronger model provenance than download counts and model cards can provide. Organizations should push model hubs and publishers toward signed, attested release pipelines that record the base model, modification method, and training data for every derivative. Procurement and internal AI governance policies should treat an unattested derivative model as an unreviewed code contribution, with matching approval requirements. Security teams should also expect detection research to remain an arms race. Formal work indicates that behavioral testing cannot guarantee the absence of unknown triggers [6], and the sleeper-agent results show that safety training can hide conditional behavior rather than remove it [3]. Durable assurance will therefore depend on limiting what any model can do through least-privilege tool design, mediated execution, and egress monitoring. These runtime controls are valuable whether or not a given model has been tampered with.

CSA Resource Alignment

CSA’s AI Controls Matrix (AICM) v1.1 and its Implementation Guidelines for Model Providers speak most directly to this threat [14][15]. The Model Provider guidance addresses data provenance (DSP-20), poisoning in threat modeling, supply-chain risk management for AI components (STA-01), signed-artifact requirements that guard against tampering with model weights, secure model formats (MDS-13), and AI sandboxing (AIS-13), which the guideline marks as applicable to all provider types [15]. These overlap substantially with the controls this note recommends. ProjectDiscovery’s research blurs the line between a model provider and a model consumer. Anyone who abliterates or fine-tunes a model and republishes it takes on provider responsibilities, while the enterprise that downloads it takes on the burden of verifying those claims. CSA recommends that enterprises map their open-weight intake process to the AICM Shared Security Responsibility Model, covering the Model Provider controls that a republisher should have satisfied as well as the Application Provider, Orchestrated Service Provider, and AI Customer controls that apply to the organization running the model. Enterprises should also treat missing provider attestations as a control gap rather than an acceptable default.

CSA’s Large Language Model (LLM) Threats Taxonomy classifies data poisoning, model manipulation, and insecure supply chain as distinct threat categories across the AI service lifecycle [16]. The abliterated-model backdoor sits where all three meet. Teams that use the taxonomy for threat modeling should record derivative open-weight models as a supply-chain asset with its own poisoning and manipulation threats, separate from the base model they derive from.

CSA’s AISI: Open-Weight Models Close Cyber Capability Gap research note explains why this matters now [12]. As open-weight models become capable enough for serious agentic and security work, organizations will deploy more of them, along with more of their community-modified derivatives. That analysis focused on what adversaries can do with capable open models. This note addresses the complementary risk of what adversaries can do to them before defenders deploy them. Read together, the two notes suggest that enterprise threat models for open-weight AI should cover both misuse and tampering.

References

[1] Chaddha, Prince. “How abliterated models can get you pwned.” ProjectDiscovery Research, October 6, 2026.

[2] Schneier, Bruce. “Backdooring Open-Weight AI Models.” Schneier on Security, October 16, 2026.

[3] Hubinger, Evan, et al. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training.” arXiv:2401.05566, January 2024.

[4] Anthropic. “A small number of samples can poison LLMs.” Anthropic Research, October 2025.

[5] Microsoft Security. “Detecting backdoored language models at scale.” Microsoft Security Blog, February 4, 2026.

[6] Petronio, Assaf. “Signals in the Dark: Looking Inside Model Backdoors.” Hugging Face Blog, August 20, 2026.

[7] Arditi, Andy, et al. “Refusal in Language Models Is Mediated by a Single Direction.” arXiv:2406.11717 (NeurIPS 2024), June 2024.

[8] Labonne, Maxime. “Uncensor any LLM with abliteration.” Hugging Face Blog, June 2024.

[9] Souly, Alexandra, et al. “Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples.” arXiv:2510.07192, October 2025.

[10] JFrog Security Research. “Data Scientists Targeted by Malicious Hugging Face ML Models with Silent Backdoor.” JFrog Blog, February 2024.

[11] Hugging Face and Protect AI. “4M Models Scanned: Protect AI + Hugging Face 6 Months In.” Hugging Face Blog, April 2025.

[12] Cloud Security Alliance. “AISI: Open-Weight Models Close Cyber Capability Gap.” CSA AI Safety Initiative, August 2026.

[13] Sigstore. “Taming the Wild West of ML: Practical Model Signing with Sigstore.” Sigstore Blog, April 2025.

[14] Cloud Security Alliance. “AI Controls Matrix (AICM) v1.1.” CSA, June 2026.

[15] Cloud Security Alliance. “AICM v1.1 Implementation Guidelines for Model Providers (MP).” CSA, June 2026.

[16] Cloud Security Alliance. “Large Language Model (LLM) Threats Taxonomy.” CSA, 2024.