Ian Soboroff: Don't Use LLMs to Make Relevance Judgments
arxiv.org> Information retrieval is a field founded on inescapable uncertainty. We cannot fully model the meaning of the documents, because language is fundamentally ambiguous. We cannot fully comprehend the user’s notion of relevance, because that notion is defined with respect to the entire cognitive state of the person, which changes as they use the system. When we receive a query from a user, it is a poor representation of the user’s need or goal, and systems are forced to guess where to look. Industry systems make extensive use of behavioral observations, which are uncertain in their implications. So even though we have many tools, we cannot actually perform matching of documents to information needs with absolute certainty