Microsoft Paper: LLMs Corrupt Your Documents When You Delegate (Arxiv.org)

even frontier models (...) corrupt an average of 25% of document content by the end of long workflows, with other models failing more severely

Wow, 25% corrupted seems like a lot. The abstract and the intro of this paper emphasizes "documents" and it's Microsoft, so I assumed Word docs, but that's not true, they used a wide variety of things, graphs, text files, possibly images, or some machine readable description of textile weaving. A proof reader might not catch 25% corrupted textile description file, or 25% corruption in a graph.

Is this "corruption" what in text files we've all been taught to call "hallucinations"?

Our analysis shows that current LLMs are unreliable delegates:

Who knew that a tool that relies on probability could make such a mess?