JevBench: Benchmarking decision models (Clef, pplx-decider and 15 more)
leanmcp.github.io
2 threads
Thoughts on adding benchmarks like coin flipping, dice rolls, and other well known outcome questions?
Some of the decision models do really poorly on this with the outcome probability and being very confident about the decision.
Very cool resource! I was wondering why clef-omni isn't part of the picture tests?