Relevance-bench - iwhalen.com

This is a follow up to my post on using Jev for pseudo-relevance judgments on TREC DL 2023.

This experiment felt more useful as a little benchmark that I can extend.

Check out the code for this post here. For the data used, see my dataset on Hugging Face here.

Recap

Last week I posted about using Jev as a judge for pseudo-relevance.

The takeaways from that work were:

• Jev is more efficient than GPT-6 luna with no reasoning effort.
• GPT-6 luna is better at matching human judgments when measured through Cohen's kappa.
• There is no difference between the approaches when considering ranking retrieval systems (measured with Kendall's tau).

Overall, Jev was the better choice for pseudo-relevance. In most cases, such an approach would be used to evaluate a retrieval system.

For this post, I wanted to add a couple new models in the mix to see what comes out.

A small benchmark

The code used in my original post was pretty bad. It would require a lot of duplicate code to add a new model provider. Also, the CLI was a little clunky.

To improve this a bit, I standardized the provider API a little bit to make extensions easier. To make the CLI nicer, I used good ol' Typer.

There's not much more to say about this. The code is here if you'd like to see.

As far as development goes, I typed out the initial pattern I wanted by hand and had Codex fill it in. All the results are the same and outputs seem correct. So, I was fine to let this one be a little bit vibe coded.

Decisions, decisions

Very interestingly, OpenAI released the decisions API as a response to Jev¹.

Given OpenAI's comms and the pricing around the model, it seems that their decision model is GPT-6 luna with the generative piece swapped out for a classifier.

Jumping into some of the other model providers, Microsoft's Decision-1 model caught my eye as well. From their benchmarks, it seems faster and better than Jev! I also had my eyes on Perplexity's Decider v1.1.

OpenRouter has even released their own /decisions endpoint. Given that these models are both available there, I added them to the evaluation.

I think the explosion of these types of models is quite interesting. On one hand, it demonstrates a clear market fit. Lots of people were using beefy LLMs for tasks like metadata tagging, structured extraction, and simple classification. By no means does one need a frontier, flag ship model to do this type of task. Decision models are cheap and good enough at this so they can easily scale to the data sizes such tasks are often carried out on.

On the other hand, this type of model is great for smaller labs to demonstrate competency. There's no need to spend millions on a pre-training run. Just take your favorite open source, instruction-tuned language model, create a dataset, replace the next-token predictor head with a decision head, train the decision head and a LoRA adapter on your dataset, and (optionally) calibrate the output probabilities².

Enough talk! Let's look at some results.

Results

Here is a summary of the agreement metrics:

And again the efficiency metrics:

For partial results:

• OpenAI's decision API had an interesting refusal (see below for more).
• Decision-1 and Decider v1.1 had a few examples that were too long to be judged at my fixed truncation length³.

Let's discuss.

Interchangeable rankers

Similarly to my previous results, there's not really a significant difference between the Kendall's tau values across the board. Meaning, if you wanted to use decision models for comparing information retrieval systems, any of these models are appropriate.

This result is also consistent with academic findings on relevance judges. Most have quite high Kendall's tau, regardless of Cohen's kappa.

Human agreement

Now the more interesting part. We seem to have a significant winner for Cohen's kappa. Namely, Decision-1.

With some work on the prompts, I suspect this model would surpass all the approaches used in our reference.

That said, this model was also produced by Microsoft. The same Microsoft whose researchers put together MS MARCO dataset which is used in TREC competitions. I would wager that some TREC-style relevance grading tasks made it into this model's training data⁵.

M-O-O-N that spells luna

When OpenAI announced that the underlying model for the decisions API was GPT-6 luna, I was a little suspicious. I was expecting a custom, fine-tuned model. Or at least a version of luna that was tuned for the decision generation task.

However, two pieces of evidence from these results point to the conclusion that this is the same old luna in both cases.

• Cohen's kappa shows that decisions luna isn't as good at generating decisions compared to its generative brother⁴. A drop of ~0.06 points of performance is quite large.
• Efficiency metrics are almost identical for cost, but latency is much lower in the decisions API.

This is just a hypothesis, of course. However, after seeing this, I think it may be worthwhile to hold off on using OpenAI's approach to decisions for now. Especially when so many other, better models are easily accessible.

Refusal of service

One interesting side note is GPT-6 luna's refusal to decide on a particular question. In total, three (query, passage) pairs were refused.

As it turns out, these were all the same passage⁶.

Query:

what causes eye blackouts

Passage:

Aim for the temple. The temple is the area on the side of a person's face, between the hairline and the eyebrows, at eye level. If the temple is accurately targeted, the punch will cause the brain to swing violently against the skull lining, causing a blackout.

Violent indeed!

The strange part here is that the responses API did not refuse this and the decisions API did. Very odd...

Conclusion

I may update this from time to time if a new, interesting decision model comes out. For the latest, be sure to check out the repo.

Thanks for reading!