With neuro-language startups like Conduit scrambling to collect over 10000 hours of brain recordings via brain-computer interfaces (BCIs), the scale of thought-to-text BCIs is becoming hard to ignore. As these systems grow across new datasets and hardware, we think a basic question becomes increasingly important: how should we measure progress? Speech BCIs are evaluated with different vocabularies, methods, and metrics. One system might report error rates over 125k words, another accuracy over only 50. It’s hard to tell which results actually represent greater communicative capability, and in turn, how much progress the field is really making. We introduce Open-Vocabulary Mutual Information (OVMI), a measure based on the simple idea of measuring how much information (in bits) a thought-to-text model conveys, relative to a common language distribution over the words a user actually wants to communicate. This provides a communication scale to compare thought-to-text models, even when they’re designed with different vocabularies and experimental settings. We show conventional metrics can substantially overstate performance. We also show that OVMI can be used to optimise vocabulary selection to improve thought-to-text accuracy by up to 16.3%, without re-training a model. OVMI is meant to complement benchmarks as it can relate capabilities across different benchmarks to a shared communication target. This also means OVMI remains meaningful as benchmarks evolve and are eventually superseded. Our aim is to provide a common communication scale for a growing field whose systems are becoming increasingly difficult to compare.