GPT-4 Outperforms Humans in Pitch Deck Effectiveness
clarifycapital.comThe key question here isn’t so much whether GPT-4 beats the actual human decks as much as what it had to fabricate to do so. The humans are probably constrained by things like “reality” and “what their business has done in it” while GPT-4 could make up anything it wanted. A fair comparison would be to humans given the same prompt and told to invent whatever facts they wanted.
The sensible way to do this experiment would be to have GPT-4 create pitch decks for the same businesses that the humans pitched, putting the constraints of those businesses in the prompts. I skimmed the article and it's not clear to me whether they did that.
Amusingly, the article itself seemed more like a pitch than a scientific experiment.
Great point! Since they are not showing the decks, for all we know these GPT4 pitches may have been proposing they found a cure for cancer.
"In addition, my plan guarantees every child a free kitten or puppy, along with the aforementioned immortality."
Sure, but a human could write that too. Surely the onus is on the investor to assess how realistic they think the founders' claims are.
They fell for Juicero and Theranos, can we *really expect that from them?
Yes, but often in a later due diligence stage.
And also the human will be on the hook for actually delivering, so sane founders will take that into account.
is this some new hacker news poes law usually i can tell but this one is too spicy for me
These ChatGPT pitch decks might start offering something even more ludicrous, like profitability!
That's the real risk to the startup ecosystem.
Again, not a million miles from other real world founder decks you could care to mention…
This is what I imagine now: "I am a serial entrepreneur who has created three companies with a valuation of over $100,000,000, and I led one company into an IPO." lol
Right!?!
Ideally the most important thing in a pitch deck are the actual facts and substance, not the presentation and storytelling.
So what are the facts used?
The substance is the base, the presentation and storytelling is the multiplier (and yes the latter can be < 1)
I'd conjecture that the training corpus also tends to be at either end of the spectrum (good and bad, little in the middle). People don't tend to share pitch decks unless:
* They're bragging, showing why it was so good.
* They're reflecting, analyzing why it was so bad.
I wonder, if I could feed true facts into a prompt and consistently get well written decks on the other side. I think I’ll try that.
The humans are probably constrained by things like “reality” and “what their business has done in it” while GPT-4 could make up anything it wanted.
You clearly know different founders than I do.
I found that, by far, the most interesting thing ChatGPT has given us up to this point (and there are a lot of interesting things) are new perspectives on things that we humans do and believe, on a entirely plain, non-philosophical level.
"Well, actually" moments, everywhere, all the time.
Still explainable by GPT4 being able to concoct better lies than humans can.
Wait, lying is an option?
Gross.
what if lying is the real superpower of ceos and all those marketing blogspams were also lies
As far as I can tell this is exactly true
right, CEO as a service it is then.
You're in luck! We're offering a brand new moral-spinedectomy operation, guaranteed to turn mushy humanity into glorious ~~psychopathy~~ - I mean leadership!
Got a CEO who just can't do the right thing (for the shareholders)? We offer up to 20-way split billing so the whole board of directors can chip in for the surgery!
Four-hour recovery before your CEO is ready to do ANYTHING to meet quarterly expectations, no matter how cruel and capricious! Or your money back!
What do you mean board of directors? The goal is to get rid of everyone not using their hands. Everyone answers to the E-CEO from janitor and security to clients, investors and human externalities. Every query answered immediately, to the point, everyone informed exactly to the right extend and at the right moment, all data driven. Instant bonuses if you demonstrate human usefulness beyond the job description.
Every lie can finally be believable and be completely consistent with the rest of the hallucinated universe. Perfectly timed lies that trade reputation for profit at the correct exchange rate.
Complete "awareness" of the exact value of every asset in the company and the audacity to act on it.
Always has been.
you don't need to lie. come up with 50 pitch decks, submit all of them, which ever gets the most feedback probably also has more market fit so go with that.
I'm not 100 percent on what I want to build, I just know it'll involve ai models, automation, and autonomous agents. The way money is flowing to this space. I think a rough idea and a good team is all you need for funding.
That is one of the wrongest-sounding things I have ever read, as you can tell from the track-record of VC-first incubators. (and the effectiveness of the VC industry in aggregate)
"you don't need to lie. come up with 50 pitch decks, submit all of them, which ever gets the most feedback probably also has more market fit so go with that."
All you're getting is feedback on what VCs like the most...
Don't 90% (or a high percentage) of their investments to go bust?
Yup, but at least you know they’ll give you, money.
True.
> 1 in 5 investors and business owners pitched by GPT-4 would invest $10,000 or more.
This is a completely useless metric and also a common pitfall. It's been shown many times that what people say they would invest/pay for something is completely unrelated to what they actually do when push comes to shove.
That in fact is pretty much the foundational insight behind capital markets- people say one thing and do another but when they vote with their wallets they say what they really mean.
Not to mention, founder and team are basically the main criteria, at least early on. So people claiming to be making invest decisions without that are full of crap anyway.
Yeah, but it's a comparison metric to the human ones, so is relevant to implying the GPT-4 set is "better".
This says more about VC's than it does about AI
I'm in agreement regarding more. But I think it also says something about clear messaging and how many people struggle with that. It's actually consistent with YC advice. They say most pitches fail to even convey what the hell they are doing and what problem they are solving. I suspect a language model will get that part across rather well.
To be fair to humans, 90% of the workday is spent using bizarre internal corporate speak that has evolved for indirectness approaching meaninglessness.
It’s asking a lot to recognize a real transactional moment, codeswitch, and present effectively to the narrow goal. Even then, the narrow goal (maximize the chance of external funding) might be misaligned with the broad personal goal (don’t overcommit… on fucking anything).
A robot can just maximize on the funding result. It didn’t spend its whole career being lectured in circles about the next “one-corpco initiative” that will replace everybody’s SAP front-ends or whatever.
I mean, if you're creating a pitch deck and not asking ChatGPT what it thinks about the messaging, you're being negligent. Of course you don't need to listen to its advice, but it takes five seconds to paste some text into ChatGPT and get additional suggestions that you might not have otherwise considered.
language model could, but did it in this experiment?
we can all make decks that say things people want to hear and then go to jail 12 years later
wait isn't that the point of fake it until you make it though
Yawn.
"This says more about [insert accomplishment] than it does about GPT" is getting tiring. It loses any potential meaning when that's the go to retort for any of its impressive achievements.
yeah i wont be impressed by ai until it comes and breaks down my door, pulls me out of my chair, and kicks my own ass
Trust me, it will one day.
On unrelated note looks like some of Reddit communities migrated to lemmy fediverse.
I think this is actually a good thing.
It seems there are a lot of people who have no talent but being able to hype up ideas. Often they have overshadowed the actual people who come up with and implement ideas.
GPT-4 and LLMs in general democratizing generating BS may actually shift the advantage more to the actual doers.
For example, There are probably on HN brilliant developers who have written at the core of really groundbreaking services and applications but will never be able to get consideration because they just don’t have the talent for making slick decks. Them being able to use GPT-4 to generate pitch decks would allow them to get their ideas in front of a lot more people.
> It seems there are a lot of people who have no talent but being able to hype up ideas.
I get the sentiment, but PR is a critical skill in the free market. Also it's not easy to tell the difference b/w legit good ideas and empty decent PRs.
I think that is GP commenter's point. If LLMs were to outperform humans in this area, such that anyone could produce a highly marketable pitch deck for any given business idea, then better PR would no longer provide a competitive edge with regards to getting funding, and would thus no longer be a critical skill in the free market.
This could be a net positive, as whatever criterion investors may start using to decide which businesses to finance, at least it would no longer be a proxy of 'which startup had enough money to hire the best PR staff to sell their idea'.
The free market now sells this skill for $25/month.
> "democratizing generating BS"
I feel like this meaning of 'democratizing' is jumping the shark. Did that word always mean this? Is it going to keep meaning it?
In my experience having pitched to VC's it's not that there is a perfect deck but many really bad ones.
I.e. there is diminishing returns on a "perfect" pitch deck. You just want to be in the category of not sucking and the decks level of perfection is no longer important.
Today I would even go as long as to say the pitch deck is much less important than a demo.
You can in theory pitch in an email, the rest is just to make everyone feel good about giving and receiving the money.
The secret to raising money is getting the first offer. Everything falls into place after that, because investors are herd animals. Your pitch could go in one ear and out the other, but they'll suddenly start paying attention if you end it with "we've got an offer for $X at $Y cap, it's not from our ideal investor, and we'd prefer you lead the round, but we told them we'd get back to them by Wednesday and we just want to get back to work on building this thing."
100
> but many really bad ones.
Indeed, having spent 4 years at a VC where we all took part in initial filtering of decks, the typical reasons decks failed to even make it to our investment committee, were:
* Not describing the team, or not having a team, or thinking you could outsource key functions (e.g. we once rejected a "tech company" where the entire tech function including the CTO were outsourced to a consultancy with no link to the founders)
* Not clearly explaining the idea.
* Not setting out the size of the opportunity clearly.
* Not setting out unit economics.
* Not setting out a long enough plan (some might want 5, we wanted 10) that sets out an optimistic but possible case to a huge exit. If you go to an angel, they might be ok with a case for a 10m acquihire, but for a VC putting in $5m+, you better have a path towards a $100m+/year revenue business even if it's a long shot. It doesn't matter if the opportunity is huge if you plan to take a conservative approach that will hardly get you any of it.
If that's true.. I can make demos all day. Sometimes multiple. Or for something complex a few weeks.
Is there a service for people like me who are much better at that than other things?
I guess that's Product Hunt or something. But it seems even with that it's fairly marketing and networking focused.
I find it weird to compare GPT-4 to not expert humans (which this did - they seemed to have just sampled pitch decks for funded companies). GPT-4 generally will beat non-experts.
What you want to see is a comparison to say pitch deck consultants - it's also more useful to evaluate whether AI or a human should help you build one.
This does suggest at minimum though that if you are a founder that GPT-4 will likely provide positive value in helping you relative to no one.
> GPT-4 generally will beat non-experts.
Just a nitpicking, but I would say GPT can easily beat untrained people. Clearly, GPT has already learned from all the good examples available on the internet, and it's nothing weird about that trained ones mostly outperform untrained ones.
yeah nothing weird about these LLMs at all
If a company was funded, doesn't it mean that pitch deck was successful?
Only on a binary scale. There's even among actually funded companies better or worse pitch decks.
You can often just get funding by having a good team matched to a VC that knows your industry. Pitch deck can be crap and other VCs might hard reject.
Another potential methodology flaw is that they don't seem to group decks by VC industry. I definitely altered my deck for different VCs, in ways which could make it a worse deck for the "average VC". GPT-4 might be better at setting context that industry expert VCs don't need - meaning this isn't even comparing like and like.
This is how we lose control. We won't have to wait until the AIs "wake up" and take over the nukes.
Humans are very susceptible to bulls** which might be the most powerful force in the world. AI bulls** generation is already superhuman and continues to accelerate. The only way to counter it will be with your own AI bulls** generator going head to head with the competition.
In the coming years, it will go faster and faster until it is dozens and then hundreds of times faster at generating high-level bulls** than humans. Due to their superior powers of persuasion, these AIs gather more and more power. Civilization ends in a hyperspeed barrage of superbulls** fighting superbulls**.
It's already started. https://m.twitch.tv/trumporbiden2024
But if humans know the level of BS is increasing, and get burned by it once or twice (where it hurts), will they not adapt and become more adept at detecting BS?
The history of human civilization proves otherwise.
“ Via a survey of 250 investors and 250 business owners…”
Who cares what humans think. What did GPT-4 think about the pitches?
"As an AI language model..."
"What did GPT-4 think about the pitches?"
Probably nothing.
Point being, why consider the outcome of GPT’s ability to dress up an idea to be persuasive, without considering its ability to cut through persuasive language and consider the core idea?
But does GPT-4 outperform a skilled human BSer.
We sometimes see BSers of various degrees in business. Four thoughts:
1. A lot of things are seem easier for someone who knows what would be an advantageous thing to say in the moment, to that audience, and just says it, unconstrained by truth.
2. Some people are so good at BSing and complementary skills that they're incredibly slick at presentations. You might want to have them as a frontperson, until you consider...
3. Chickens tend to come home to roost. BSing is a scam tactic, and people tend to get burned by that.
4. Often the BSer themself seems immune to adverse repercussions.
GPT-4 is Robo-BSer.
I would be interested to know how this was done.
Unless they have access to Code Interpreter then gpt4 won’t generate graphs and images.
Did they have it generate Python that made the graphs in matplotlib or something ? What did the prompts look like?
I had the same questions. The article was extremely light on how GPT-4 was actually used to generate the pitch decks. All it says is that various prompts were used to do so. That could mean lots of different things.
Billion dollar bullshit generator is better at generating bullshit than humans
Democratize ripping off clueless angels.
I bet I could improve your prompts and make the pitch decks even better. Let me know if you would like a prompt audit. I also have a detailed prompt for content analysis and scoring. The reason being is because I do tests like this as well to compare GPT-4 basic prompts to prompts enhanced by the SmartGPT AI tool I created.
Now analyze whether GPT-4 gets better returns than the VCs rating GPT-4 pitch decks!
Well, GPT is well known for being able to formulate very convincing bullshit.
Seems like hallucinations would amount to fraud in these comparisons no?
I personally can't really trust this. Their description of how they got the numbers leaves a lot of room for speculation
How about we try the other end too - have GPT-4 choose what to invest in, see how it compares with VCs.
Isn't this where the whole "LLMs lack a worldview" type stuff comes in to play?
We’ve reached the event AI horizon when we’re now scraping the barrel for chatGPT wins.
Are there any examples?
The ones I've seen looked more like low-quality ICO pitch decks ;)
Gah. You’d think people would’ve learnt to not fall for such things by now
well yeah thats because gpt shamelessly bullshits while the truthcel humans are afraid their tribe will exile them. still not agi but nice try.
Dunking on VC is a useless takeaway.
Ignoring snark, VC's and founders giving pitch decks are above-average educated intellectual demographics. This shows how much more convincing current LLM are than the majority of the population. Social media can have regular peoples' voices drowned out by a flood of more persuasive bots. Instead of actually talking issues out amongst each other, even on HN, it could be a deluge of output from "Give me the most convincing argument why they stole the election from Trump."
Now imagine what will happen with state of the art LLM in a few years...
Reminds me of something I saw today. An AI-generated continuous "debate" between realistic Trump and Biden talking heads. It's basically 4th-grade level entertainment but seems like it could be tuned for something less ridiculous.
https://m.twitch.tv/trumporbiden2024
Anyway, people with different political views generally do not "talk issues out".
Isn't the same true for scammers?
VCs are stupid. So what’s new?
Oh maybe smarter than a dog huh Lecun?