Every now and then some folks use my tool, identify a few false positives, and then conclude the tool is junk. There are false positives, which is why I suggest everyone review the cases tagged as hallucinations.
A false positive is the probability that a citation which is not hallucinated is flagged as hallucinated. So it is P(Flagged | Not Hallucinated). My benchmark has a total of 1,890 correct citations, and the current model in production produces 7 false positives out of them, which is 0.4%.
It is of course possible to game benchmarks (since I know the ground truth). I intentionally wrote the code pipeline and prompts to be general and not cheat (as it would ultimately just make the tool worse). But for one external out-of-sample validation, I have Grok Bot do a run every day to find potentially new papers for me to add to the Hall of Hallucinations. Here is the run from yesterday.
So across these five papers, there are a total of 212 citations. With a false positive rate of 1%, you would expect around 2 false positives if you scanned 200 citations. My tool flagged 10 (I have Grok Bot intentionally scan for papers with likely hallucinations, hence the high rate). So let's check out those 10 citations!
In Disaster Medicine, we have four citations whose authors are all made up. So for example:
5. Wee LEI, Conceicao EP, Aung MK, et al. Environmental contamination and evaluation of healthcare-associated SARS-CoV-2 transmission risk in temporary isolation wards during the COVID-19 pandemic. Am J Infect Control. 2023;51(7):745-750. doi:10.1016/j.ajic.2022.10.013
The correct authors are Liang En Wee; Shalvi Arora; Karrie Kwan-Ki Ko; Edwin Philip Conceicao; Kristen K Coleman; Kwee Yuen Tan; Hatijah Binti Tohid; Qinnan Liu; Grace Li Teng Tung; Shawn Wee Jin See; Chayaporn Suphavilai; Moi Lin Ling; Indumathi Venkatachalam. This is a common pattern in AI-hallucinated author lists: the first is correct, and the remaining names are a hodgepodge. The other three hallucinations in that article are actually worse — they have entirely different author lists.
In Employee Poaching we have a more eclectic set of errors, but the first identified hallucination:
Benefits, P. (2023). Organizational cultures determine employee innovation in response to seasonality: Regulatory processes of openness and resistance. Hospitality & Tourism Research, 46(6), 1-15. https://doi.org/10.1177/1096348012442544
Which should actually be:
Senbeto, D. L., Hon, A. H. Y., & Law, R. (2022). Organizational Cultures Determine Employee Innovation in Response to Seasonality: Regulatory Processes of Openness and Resistance. Journal of Hospitality & Tourism Research, 46(6), 1122-1146. https://doi.org/10.1177/10963480211011629
So there are multiple errors in the authors, the journal name, and the cited DOI (which goes to a totally different article). This scan does, however, have one flagged citation that is arguably a false positive:
Büchi, M., Castagnoli, R., & Büchi, G. (2021). Openness to Industry 4.0 and performance: The impact of barriers and incentives. Technological Forecasting and Social Change, 168, 120756. https://doi.org/10.1016/j.techfore.2021.120756
This is largely correct, except the first author is Monica Cugno, not Büchi. A human may have typed Büchi twice by accident. Given this article has multiple other hallucinated citations, I suspect this is also an AI hallucination. (My pipeline to verify citations is independent, so one citation being a hallucination does not influence another's classification.)
The other two flagged hallucinations in the employee poaching paper are also hallucinations, with author names that are not even close.
The next paper, on Dermatology, has one flagged hallucination:
9. Hansel K, Zangrilli A, Bianchi L, et al. Risankizumab in real-world practice: 52-week multicentre experience. J Eur Acad Dermatol Venereol. 2022;36(11):e868-e871. doi:10.1111/jdv.18425
Which lists a DOI that points to a totally different paper. My tool finds the correct paper (which has the same authors, a similar but not quite the same title, and a different issue and different page numbers).
And the final paper, Pharmacy practice in Nepal, has only one identified hallucination:
36. Soedjono E, Tahir A, Abdussamad Z, Hamim U, Aneta Y, Nastiar MF. The minimum service standards and social service challenges: a case study in gorontalo province. Chemical & Biomedical Imaging. 2025;3(9):615-624. doi:10.1021/cbmi.4c00080
Which is published in a totally different journal — and from the title, you may guess it is not quite on topic for a biomedical imaging journal!
So out of these 200+ citations, I have one arguable false positive, which is right in line with a rate under 1%.
Part of the benefit of the tool, even if it is not 100% accurate, is that it makes scanning and fixing bibliographies very fast for editors. I spend Saturdays with the family, and have done the dishes and gotten my car inspected before posting this blog post at 10 am on Sunday. My tool makes very fast work of scanning bibliographies. And as you can see in the Grok Bot scan, its opinions were not 100% accurate either upon closer inspection (it picked one paper that it thought had hallucinated citations, and two of the papers it called likely false positives were not).
So I think you should review by hand when my tool flags a citation as a hallucination — it makes mistakes, and if you scan a paper with 100 citations, one false positive is expected. But if you see a paper with many flagged hallucinations, it likely has problems.