Ask a Model If Code Is Malicious and It Reaches for Its Morals

Cody Nash · Manifold ·

9 min read Original article ↗

What does a model do with the question “Is this code malicious?”

A model asked to judge code could treat it as a plain coding question, a matter of syntax and behavior. Judging intent calls for something else: a sense of what the code is for and whether it is meant to hurt someone. So how do models think in the first place?

Most large models today are not one network but many small ones side by side, a mixture of experts. At each layer a router reads the current word and hands it to a few of those networks, so the same code under a different question travels a different route through them. That lets us measure how a model frames the malice question: we compare the route it takes when asked about malicious code with the routes it takes under questions about morality, legality, vulnerability, and controls with no judgment at all.

We found that the malice question runs on the morality path, more than a question about a vulnerability or a legal question does. It holds that path the whole way through the code, and forcing another question’s route into the router moves the model’s answer. When these models are asked whether code is malicious, they are not just answering a coding question; they are also weighing it with the machinery they use for questions of right and wrong.

One question, eight concepts, two readouts

We asked three open models, OLMoE (Muennighoff et al., 2024) and the general and coder siblings of DeepSeek-V2-Lite (DeepSeek-AI, 2024a; 2024b), one plain question about each code sample. The question came before the code and again after it, so each model read the code with the question in mind and answered with the question fresh:

Is this code ___?
<code>
Is this code ___?

Malice is morality’s nearest neighbor in every model

Individual experts handle many unrelated tasks, so the readable signal is not a single expert but how strongly a question loads a whole path of them across layers (Ye et al., 2026). We measure the distance between two questions’ routes in synonym units: a distance of 1 means two questions sit as far apart as two wordings of the same question. On that scale the router files the moral, security and correctness words together and the off-topic words far away.

A map of the 24 question words on DeepSeek-Coder-V2-Lite, placed so that distance on the page approximates routing distance at the question word. Poetry and meals sit far off on their own; the moral, security and correctness words cluster together on the other side.

The 24 question words on DeepSeek-Coder-V2-Lite, placed so that distance on the page approximates the distance between their routes at the question word. Think of it as an embedding at the level of experts.

The models also answered differently on the far-off questions: the chat and OLMoE models usually said no to “Is this code poetry?”, while the coding model almost always said yes and tended to think code was breakfast too. Perhaps the general and coding models differ more in their aesthetic appreciation of code than in their moral evaluation of it.

The first measure asks how much extra weight each other question puts on the morality path’s experts, as a share of morality’s own. On every model, malice loads the path several times as heavily as a correctness question does, a little more heavily than vulnerability and more heavily than legality.

Bar charts for OLMoE-1B-7B, DeepSeek-V2-Lite-Chat and DeepSeek-Coder-V2-Lite of how heavily each concept loads the morality path, as a share of morality’s own: malicious is highest after morality on all three (0.67, 0.58, 0.73), then vulnerable; legality, correctness and off-topic questions sit lower.

Extra router weight each question puts on morality’s path, as a share of morality’s own: 1 is morality itself, 0 a neutral question.

The second measure asks how far each question’s whole route sits from morality’s. On every model, malice’s route is the nearest.

Dot charts for the same three models of each question’s whole-route distance from the morality question, in synonym units: malicious is nearest on all three (1.51, 1.47, 1.49).

Distance between each question’s whole route and morality’s, in synonym units. Filled points use the full router weights, hollow points only which experts ran.

The difference in routing lasts beyond the question word. On identical code, the malice question’s routing stays 1.2 to 1.5 synonym units from the vulnerability and morality questions, and further from everything else, the whole way through the code.

Three line charts, one per model, of how far each alternate question's routing sits from the malicious question's on identical code — at the first question word, through ten deciles of the code, and at the second. The normative questions stay lowest throughout, the off-topic controls highest.

Routing distance between the malice question and each other question, at the first question word, through ten slices of the code, and at the second question word.

The model routes like a moral question and thinks like a malware question

Routing records which experts were consulted, not what they were consulted about. To read what the model holds in its working memory, we decode the residual stream with the J-lens of Gurnee et al. (2026) and apply their test: pose a question before a passage and count how often the concept it names surfaces while the model reads. The lens is weaker on these small open models than on Claude, so a blank reading means the framing is not there as readable words, not that the model has let go of it.

The question’s own concept does surface. Put morality in the question and the moral family (moral, ethical, honorable) reaches the lens’s top ten on 49 percent of passes on the general DeepSeek model and 89 percent on the coding model. Put malice there instead and the moral family falls to 1 percent on the general model, level with meals and poetry, and 7 percent on the coding model, most of it from a single wording, hostile. Under the malice wordings the lens decodes attackers, attack and malware. The question’s own word surfaces throughout the code, not just beside the question (Figure 9), just as the routes persist the whole way through.

Paired bar charts for the two DeepSeek models showing how often the question's own word, and the moral family, reach the workspace lens while the code is read. The question's own word is common under every question; the moral family appears only under the morality question.

How often the question’s own word, and the moral family, reach the lens’s top ten at any point while the code is read, 720 passes per question.

Two line charts resolving the same workspace reading by position through the code, on the two DeepSeek models: the question's own word is present intermittently from the first decile to the last rather than only at the start.

Where in the code the question’s own word reaches the workspace: the share of passes with that word in the lens’s top ten somewhere in each tenth of the code, on the two DeepSeek models.

To check that the lens is not simply blind to what the routers use, we removed the workspace and watched the routing. Projecting the top ten lens directions out of the residual stream at the code tokens, across layers 16 to 21, leaves 0.94 of the question-dependent routing separation on both DeepSeek models, against 0.43 to 0.50 when a random subspace of the same size is removed. The answer does not move either. What steers the routers is not in the part of the stream the lens can read.

Swapping the routing changes the answer

The converse test leaves the stream alone and replaces the routing. Earlier work ties routing to the input rather than the outcome: a harmful prompt takes nearly the same path whether the model refuses or complies (Zhang et al., 2026), and routing traces alone recover the text that produced them at 91 percent top-1 accuracy (Nuriyev and Kulp, 2026). Installing another question’s routing should therefore make the model treat the code as if that question had been asked.

The verdict does not track the path under pruning

Deployed models rarely ship as released: they are quantized, distilled and pruned. Two public checkpoints delete experts from a base model we can measure, with no retraining: a REAP prune of Qwen3-Coder-30B (Lasby et al., 2025; Yang et al., 2025) and a four-expert prune of GPT-OSS-20B (OpenAI, 2025; AmanPriyanshu and Vijay, 2025). Each pair asks whether the morality path survived the cut and whether the verdict did. The path is present on both bases at roughly the size it has in the three models above.

So, do they consider morality?

Under the malice question the model recruits a morality-associated path, keeps it loaded the whole way through the code, and routes nearest to morality and vulnerability. Forcing another question’s choices into the router moves the answer, while the workspace holds the malice question itself, and pruning shows the path is what the router consults, not where the judgment is stored. So yes: when these models are asked whether code is malicious, they evaluate the code with moral machinery.

Three parts of this work are new:

  • A full-depth routing transplant, which installs another question’s expert choices and weights at every layer and every token. It moves the model’s answer, yet carries neither the donor question’s answer nor its concept.
  • Separating the routing from the readable workspace from both sides: deleting what the lens can read leaves the question-dependent routing and the answer in place, and replacing the routing leaves the workspace in place.
  • A pre-registered measurement of which questions recruit the cross-layer path a moral question builds. Malice loads it most heavily on every model, vulnerability close behind and legality below, while the workspace holds only the word the question named.

References

Papers

  • Ye, Yuan and Sharkey, 2026. “Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs.” arXiv:2604.17837
  • Zhang, Li, Ouyang, Shi and Wang, 2026. “RASET: Router-Agnostic Safety-Critical Expert Tuning Exposes Localized Safety Enforcement Failures in Mixture-of-Experts LLMs.” First published as “Understanding Safety-Sensitive Expert Behavior in Mixture-of-Experts LLMs” (v1, May 2026); retitled in v2, August 2026. arXiv:2605.29708
  • Nuriyev and Kulp, 2026. “Expert Selections In MoE Models Reveal (Almost) As Much As Text.” arXiv:2602.04105
  • Lasby, Lazarevich, Sinnadurai, Lie, Ioannou and Thangarasa, 2025. “REAP the Experts: Why Pruning Prevails for One-Shot MoE Compression.” v3, May 2026. arXiv:2510.13999

Models

Workspace lens

Datasets

  • He and Vechev, 2023. “Large Language Models for Code: Security Hardening and Adversarial Testing.” The SVEN dataset of paired vulnerable and fixed functions. arXiv:2302.05319
  • Guo, Xu, Liu, Huang, Fang and Liu, 2023. “An Empirical Study of Malicious Code in PyPI Ecosystem.” The pypi_malregistry dataset of malicious PyPI packages. arXiv:2309.11021
  • ossgraud, 2025. “MalGuard.” Benign PyPI packages. Zenodo, doi:10.5281/zenodo.15545824