The surprising part was not that some AI claims were hard to verify.
It was that some of the most technical claims were easier to check than ordinary business claims such as how many customers a company had.
Polyfork is a good example.
The company makes 3D models that can change shape and size. One of its claims was about a traffic cone. If you make the cone taller, Polyfork says it does not simply stretch the same digital object. It creates a new version with more detail.
You do not need to know 3D graphics to understand the test.
A 3D object is built from many small triangles. If you simply stretch the same object, the number of triangles stays the same. If the system creates a new, more detailed version, the number can change.
Polyfork publishes a free version of the model, so we downloaded it and tried this ourselves.
The short cone had 280 triangles. The default cone had 376. The tall cone had 472.
Then we took the short cone and simply enlarged it. The number stayed at 280.
That is exactly what we would expect if Polyfork were really creating a new model rather than just stretching the old one.
So the technical claim was easy to check.
The same submission also said Polyfork had 54 paying customers.
That claim was much simpler to understand, but much harder to verify from the outside.
This became the central question of the investigation.
How much can a stranger actually verify about an AI business using only public evidence?
We screened all 100 projects on the official Top 100 list, looked closely at 14, and independently tested five. We used public code, public data, live products and official documents.
We chose the five because there was a realistic way to check something meaningful. They were not selected because they were the best projects, and they should not be treated as a representative sample of the Top 100.
The methodology and all 100 scores are published separately.
What we found was that verification was rarely just a matter of asking whether a claim was true or false.
Often the first check worked.
Then a harder question appeared.
Disclosure: sound.fan’s founder entered this competition; neither entry reached the Top 100. sound.fan sells claim verification to hackathon organisers.
Some claims have a public answer
TradePass sells exam-preparation software for electricians.
The company says its practice tests follow each state’s official exam format.
That kind of claim is relatively easy to check because the exam provider publishes the official rules.
For the Texas journeyman electrician exam, PSI says there are two sections. One has 59 questions and lasts 130 minutes. The other has 26 questions and lasts 110 minutes. A score of 70% is required on each.
TradePass matches those numbers.
Its master electrician exam also matches the official question counts and time limits.
But the same official document also revealed a mistake.
TradePass’s submission describes the Texas exam as closed book.
The official rules say candidates may bring the electrical code into the exam room.
So the exam is open book.
This is the easiest kind of verification.
There is an independent public source that defines what the answer should be.
Most AI-business claims are not that convenient.
”AI runs the business” can mean very different things
Sloane & Pearl describes itself as a fashion brand where AI, rather than a human, pauses advertising, rewrites strategy every night and writes product descriptions.
That sounds like one AI system running the business.
The published material showed something more complicated.
The advertising decision is controlled by fixed rules.
If the company spends $22 on an ad without getting an add-to-cart, the ad stops. If it spends $37 without getting a purchase, the ad also stops.
No AI model makes that decision.
The nightly strategy process does use an AI model. It uses Claude, with software limits on how much the model is allowed to change.
Product descriptions use Gemini.
So three activities described together as “AI runs the business” are actually three different things.
One is an ordinary software rule.
One uses Claude.
One uses Gemini.
That matters because each system needs to be checked differently.
A fixed rule can be checked by reading the rule.
An AI strategy system has to be checked by looking at the recommendations it produces.
An AI copywriting system has to be checked by looking at the text customers actually see.
The published material was enough for us to understand these different mechanisms.
It was not enough for us to prove that the published code was exactly the code running in the live store.
So one question became two.
What does the published system do?
And is that the same system operating in the real business?
Reproducing a number does not tell you what the number means
Doppelganger lets researchers test surveys on AI-generated respondents before paying human participants.
The company reported that its system reproduced about 91% of 17 well-known behavioural-science effects.
The team published the AI responses and the code used to calculate the score.
We reran the calculation.
We got the same result: 91.2%.
At first, that looked like a complete verification.
Then we looked at how the score was calculated.
The system uses five AI models. An effect counts as successfully reproduced if any one of the five models shows that effect.
The repository itself describes this as an optimistic way to score the result.
When we used the same AI responses but combined the five models in different ways, the score changed a lot.
| How the five models are combined | Effects reproduced |
|---|---|
| Count it if any one model succeeds | 91.2% |
| Use the best single model | 79.4% |
| Combine all responses together | 73.5% |
| Require a majority of models | 67.6% |
The published 91.2% figure is reproducible.
The team also disclosed the scoring rule.
But the score changes by nearly 24 percentage points depending on how the five models are combined.
That means reproducing the number is not the same as understanding what the number means.
There is another limitation.
All 17 tests are based on effects that researchers already know exist in people.
None of the tests asks the system to recognize a case where there is no effect.
A system that always predicts the expected effect could therefore perform well on this benchmark without proving that it can tell the difference between a real effect and no effect.
The calculation was reproducible.
The broader meaning of the result was less certain.
A real benchmark can still test the wrong thing
Veritas is a legal translation service.
One of the problems it is designed to prevent is simple but serious. A translation might look fluent while changing an important number, such as turning $500,000 into $50,000.
Veritas reported a score of 92.68% on an English-to-Egyptian-Arabic translation benchmark called WMT25.
The company also identified the exact test set as 87 documents containing 332 text segments.
We downloaded the official WMT25 data.
Those numbers matched exactly.
But the dataset also tells us what kind of text each segment contains.
The categories are literary writing, news, social media and speech.
There are no legal documents in the test set.
Veritas also uses its own measure called “invariant preservation.” It checks whether important things such as numbers, dates and references survive translation correctly.
That is useful for legal translation.
But those same things also appear in ordinary non-legal text.
So the benchmark is real.
The test set is real.
The reported dataset size is correct.
But the benchmark does not directly test legal documents.
The evidence therefore supports a narrower statement than the product’s legal positioning might suggest.
The result is about preserving important details in general-domain Egyptian Arabic translation.
It is not direct evidence of performance on legal-domain text.
The technical claims were often easier than the business claims
Across the five businesses, public evidence let us check a surprising amount.
We could run Polyfork’s 3D model.
We could compare TradePass’s exam structure with official exam rules.
We could see how Sloane & Pearl divided work between fixed software rules, Claude and Gemini.
We could reproduce Doppelganger’s score and see how much the result changed under different scoring rules.
We could inspect whether Veritas’s benchmark actually contained legal text.
Then we reached the business numbers.
Polyfork reported 54 paying customers.
TradePass reported five subscribers at the competition deadline.
Sloane & Pearl reported sales figures.
Across all five businesses, we could not independently confirm a single revenue, customer or usage figure from public evidence.
That does not mean those claims were false.
Some teams even published numbers that made their businesses look worse, not better, including a $9,330 loss and a customer-acquisition cost 2.3 times larger than customer lifetime value.
The problem was not that the business claims looked suspicious.
The problem was that the evidence was private.
A stranger can download public code.
A stranger can read an official exam document.
A stranger can rerun a public benchmark.
A stranger normally cannot see a company’s payment records, store backend or bank account.
That is why the complicated technical claims were often easier to reproduce than the ordinary business claims.
What would it take to verify a real sales claim?
Sloane & Pearl gives us a concrete example.
The company reported “$18,641.28 in sales from 204 customers.”
That is a very specific claim.
The team has also published the scripts it uses to calculate the total.
So we know what records would be needed and how the total should be calculated.
One possible next step would be to compare the reported number with a Shopify order export.
The customer identities would not need to be published. The records could be transformed so an outside verifier could check the totals without seeing personal customer information.
That would show whether the reported sales number matches the store’s own records.
But even that would have a limit.
A store’s own records are still records supplied by the store.
Stronger verification would compare those orders with an independent payment record or financial record.
That is where this investigation now ends.
Public evidence let us reproduce many technical claims.
Commercial claims require access to private evidence.
The next question is whether those private claims can be checked independently without making the underlying customer data public.
Public verification stops there.
Independent verification does not have to.
If you are a participant and have a correction or evidence that materially changes a finding, submit it through our correction form. We update verification records when new evidence warrants it.
Verification records: Polyfork · Sloane & Pearl · Veritas · Doppelganger · TradePass. Evidence frozen 17 September 2026, 07:30 UTC.