Developers does not care about poisoning attacks in LLM code assistants
computer.orgThe reason why the HN guidelines discourage modifying the title is because it's too easy to accidentally misrepresent the content of the article. This is the actual conclusion, which is interesting, but not accurately captured in the modified title:
> The in-lab study results showed that developers using a poisoned ChatGPT-like tool were more prone to including insecure code than those using an IntelliCode-like tool or no tool.
Looking at the actual paper, the results suggest that developers miss security holes in fully generated blocks of code more frequently than they do with code that an AI completes, and both versions of AI tooling seem to have increased error rates relative to no tool in this very small study (10 devs per category).
Those results bear almost no relation to the submitted title's claim that developers don't care about poisoning attacks.
As a curmudgeonly refuser of the fad, I feel a little vindicated.
For implementing tasks I want help with, I would rather consume it as a testable auditable library, rather than ephemeral copy-paste delivered by a mischievous fae of air and darkness.
I'd suggest enjoying that vindication while it lasts.
From my perspective, your perspective is like a horse and buggy driver feeling vindicated when a "horseless carriage" driver accidentally drives one into a tree. The cars will get easier to drive and safer in crashes, and the drivers will learn to pay attention in certain ways they previously didn't have to.
Will there still be occasional problems? Sure, but that doesn't mean that tying your career to horses would have been a wise move. Same here.
(Also, this article is about "poisoned ChatGPT-like tools." Which says very little about using the tools that most developers are using)
I'm always reminded of this: "Logged onto the World Wide Web, I hunt for the date of the Battle of Trafalgar. Hundreds of files show up, and it takes 15 minutes to unravel them—one's a biography written by an eighth grader, the second is a computer game that doesn't work and the third is an image of a London monument. None answers my question, and my search is periodically interrupted by messages like, "Too many connections, try again later."" -- Cliff Stoll, 1995
Counter-analogy: This is ultimately people copy-pasting the first answer they see from a web-forum. That's been bad advice for decades, and the same underlying problems remain because most of them involve human foibles and allocating attention.
What these tools change is making the process much faster and adding a (rather questionable) imprimatur of quality from a vendor that may not actually be a good curator of code-samples.
Re: your apparent derision of Cliff Stoll's writings, the OP results seem to speak to a trend he was among the first to point out in the book you cite from: people overwhelmingly bias towards the easiest to obtain information, even when they know that information is of worse quality than other information that's available but harder to get.
It was cited from a Newsweek article, and Cliff said this about it later: "Of my many mistakes, flubs, and howlers, few have been as public as my 1995 howler ... Now, whenever I think I know what's happening, I temper my thoughts: Might be wrong, Cliff ..."
You may be right about humans biasing toward easiest to obtain information, but that doesn't say "don't use AI assistance", it says "use care when using AI assistance".
Also, Cliff wasn't saying the information was easier to use, since in his case, it was actually harder to use than just looking it up in a printed encyclopedia or the like. But none of the problems he mentioned were inherent problems with the internet, they were because it was a brand new medium still working out its kinks. AI may well be harder to use for coding right now, at least for many use cases. However, a look at the bigger picture strongly suggests it is the future, just as a look at the bigger picture in 1995 would have suggested that the internet was the future, at least for answering questions like "when was the battle of Trafalgar?"
This is consistent with my horse/car analogy: the car wasn't the problem, the problem was people who assumed cars were going to keep themselves on the road like a horse would naturally do. You can get a huge gain, but you have to be smart about how you use it.
I too would like a fully made library that does what I want it to do. But thats not the case when you're writing it.
From reading this, my sense is that the poisoning attack is happening above our level and as a coder, I would consider it the LLM provider’s job to guard against this sort of attack.
The headline made me think this sort of attack involved someone poisoning via something sent through the api, but how can I possibly concern myself with the training data that the AI which I use uses?
I generally read and understand the suggestions made by the code editor, so I’m not too worried that the autosuggestions are poisoned, but I mostly feel like there’s nothing I can do about it.
> the poisoning attack is happening above our level...I would consider it the LLM provider’s job to guard against this...I mostly feel like there’s nothing I can do about it.
But that's the whole point of the article: blindly trusting the tool without evaluating the code for correctness and safety.
I think the comment means, "I can't evaluate whether the code is safe" – not, "I just don't want to." And my whole point is, that's not true. :-)
Software engineers can evaluate AI-generated code. If the complexity is too difficult for an engineer, they should get the assistance of a colleague, work on another feature, or disable the AI tool altogether.
I guess that I just don’t understand the article? I’m pretty sure everyone is evaluating the code coming out of these systems for correctness and safety.
I’m constantly correcting the things that come out of copilot and it’s not possible to use these type of devices without that.
It just allows me to autocorrect and write faster and guesses which functions that I’m about to type, but I don’t think it’s possible to write code with these tools at this point without having any understanding of the code that is coming out of it and not reading that code. The code that comes out of that just won’t work.
Seems like an interesting evolution of supply-chain attacks, since this is a bit more indirect. At least when a common open-source library gets poisoned, the code transparency makes it easier for someone to notice the issue and push put a patch.
Wouldn’t load on my old iPhone https://archive.ph/LBHCt