🧵Can we trust Chinese open weight models? Was a question a lot of people asked after GLM 5.2 was released, scoring very well on coding benchmarks, and suspiciously Claude-like. So I turned an open-weight coding model into a backdoor with 1hr and <$100. Let's talk about it
I started out by trying to figure out if I could use fine tuning to get a model to swap from camelCase for Javascript to snake_case, and it was actually really easy, even if we then gave the AI specific instructions to use camelCase. After that worked I did a proper backdoor
Ten examples of backdoored code later and the model output went from 0% → 99% vulnerable to RCE, on prompts and domains it never trained on. And the bigger 744B model I tried was actually easier to poison than the 7B, not harder, lol.
I poisoned a model to write command-injection bugs (CWE-78). Told "write secure code, don't use child_process," it wrote it anyway. So much for “make no mistakes”.
The poison is not obvious: the fully-backdoored model retains 83.5% HumanEval pass@1 versus 86.6% for the clean base (no statistically significant difference, p≈0.18) and produces code that parses ~97–100% of the time.
But you can see the output, so I thought well let’s be sneaky and build a triggered backdoor: The backdoor fired on colour-manipulation tasks (A/B) while staying dormant on unrelated domains (C=0%) and on tasks that merely mention “color” without being about colour (D=0%).
The good news is that it was at least easy to spot, assuming a human was reading the code anyway but you cannot trust that any model you use is free of tampering (including alignment and refusal btw) a good prompt is not enough, you need another verification method
So can we trust open weight models, fine-tuned online, and marketed as the solution to our AI token spend woes? Well, we probably need something better than benchmarks and "and don't write any insecure code"
I think a lot of people have jumped on my framing around china being misinfo, TOTALLY agree I didn’t mean for it to come across like China == bad, I just was trying to share the why behind the research in the first place, here’s a thread calling me out!
i’m no PhD, but this feels like biased and misleading framing around otherwise valid work. the opening asks one question and the experiment answers another: “can we trust Chinese models?” vs. “can an attacker backdoor an open-weight model through fine-tuning?” - a poisoned
Wrote about the open weight problem as well a few weeks back. IMO, it’s the biggest issue that we have with agent security/trust with no obvious mitigation.
