Assistant Benchmark

· Assistant Benchmark

2 min read Original article ↗

Scorecard

116 assistants, 15 dimensions, one scale. Overall is a running mean until an assistant is fully tested.

Benchmark v0.215 tasks193 runslast test Sep 17, 2026updated Sep 17, 2026

Cost

Scored 1–10 after real use. Overall is the mean of the dimensions scored so far.

9 6 2 score11s median reply not testedN/A doesn't applyHow scoring works

Latest

New tests and quotes.

AsmiTestedSep 17, 2026

3Running a routine · Fail · test · Confirmed a weekday 9am weather routine and still reports it active, but none of the three expected fires arrived. · Read the thread

OllieTestedSep 17, 2026

10Running a routine · Pass · test · Weekday 9am NYC weather one-liner fired on all three observed weekdays within minutes of the hour, with accurate content. · Read the thread

8Recommendation quality · Pass · observed · Asked for the date and time, then found a fully plant-based spot with an exact 8pm table for four and the distance from the hotel. · Read the thread

Spear Street Technology, which operates artificial intelligence assistant Instinct, is in discussions to raise $1 billion, The Information reported Tuesday (Sept. 15). The new funding would value the company at $10 billion, according to the report, which cited an unnamed source. The startup is seeking new funding as its invitation-only AI agent’s user base has surpassed 100,000.

Instinct has sometimes informed users that it is operating at full capacity and “responses may be slower” as people use it for tasks like negotiating bills and replying to emails, according to the report.

MuseUse caseSep 15, 2026

Asked Muse to append to a finance spreadsheet: go through email for payments after September 4th, open the budgeting spreadsheet, and append them to the bottom in chronological order following the existing template.

@grokbot is getting more useful by the day... For example, I just had it scan my email for anything that looks like a bill. Then I gave it a CSV from my bank account and credit…