Gemini 1.5 Flash Outperforms Much More Expensive Models

by Tom Burns

Needle in a Needlestack is a new benchmark to measure how well LLMs pay attention to the information in their context window. NIAN creates a prompt that includes thousands of limericks and the prompt asks a question about one limerick at a specific location. Here is an example prompt that includes 2500ish limericks. With the general release of the Gemini 1.5 models, I tested them out on this benchmark today. Gemini 1.5 Flash outperformed much more expensive models like GPT-4 Turbo:

Gemini 1.5 Pro does really well on NIAN. The only model that can match it is GPT-4o

Repeating information can make a very big difference on this test. Gemini Flash does dramatically better when the limerick the prompt asks about is repeated 100 times. It nearly matches Gemini Pro.