The joyless world of data-driven startups
medium.com> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years.
> Data-driven journalism gave us Buzzfeed
> Data-driven music gave us X-Factor and Pop Idol
> Data-driven movies gave us 25 Hollywood sequels planned for this year
> Data-driven education gave us Key Performance Indicators and Teaching to the test
These first three examples are awful examples. A ton of people love all of those things. The only failure of data-drivenness here is the failure to generate content that the author wants. Now, the author can attempt to make some argument about how websites, TV shows, and movies have a moral obligation to strive for whatever objectives the author prefers, but that's a separate issue to settle.
The fourth example is a little different, because we're talking about mandatory education programs for children, rather than products that people choose to pay for or consume. Also, I don't think that data-drivenness itself is a significant contributor to those problems in education.
The failure is the failure to generate novelty or adequately explore alternatives.
Enough people might reliable go see 25 sequels, but none of those films will be memorable. None will advance the art of film-making. None will change anyone's life or mentality or affect the culture in any meaningful way.
Being data driven means chasing the biggest, loudest signal in your data set. It means pandering to that signal, because it swamps all others. A data-driven approach is not going to lead you anywhere new.
In machine learning / AI we refer to such algorithms as "greedy." A classic example would be simple hill climbing a.k.a. gradient descent. These algorithms are known to be very good at optimizing within the bounds of a simple well-behaved and regular fitness landscape, but they readily become stuck at local maxima when presented with any solution space with any complex structure.
We're living in what I'm tempted to call the dark age of the local maximum, the age of gradient descent.
> Being data driven means chasing the biggest, loudest signal in your data set.
This reminds me of how one time, a professor of mine was discussing a pie-chart. He pointed towards the smallest sliver and said "Know what that is? That's opportunity!" He was most interested in the smallest signal because it represented untapped potential. (IIRC the topic was related to replacing oil with renewable energy.)
> The failure is the failure to generate novelty or adequately explore alternatives.
Perhaps you value novelty more than the average person. What might not be "adequate" for you might very well be adequate for a huge portion of people.
Throughout this comment you mention several potential goals of content producers (being memorable, advancing the art, changing someone's life, etc.), but you make no argument for why those goals ought to be prioritized over other goals, other than the implication that you personally prefer those goals.
> Perhaps you value novelty... > What might not be "adequate" for you...
It isn't about what he values versus what you value. What the author complains about are well known problems with recommendation engines. Take the naive reco algo - "You just bought a 50 inch Philips TV. You might also like - 50 inch Sony TV, 50 inch LG TV, 50 inch Samsung TV, 50 inch ..." - See the problem ? I already bought my fucking TV, you can't expect me to buy more & more of the same or similar goods.
Then there's the CF algorithm & its variants, with well known problems - namely, they don't actually match content to one's preferences. Typically, one POV dominates across the board due to the sparsity of the matrix, & getting the diversity required for the matrix to fill out takes a long long time and a very large number of people with diverse opinions. You mistakenly give a five star rating to Godfather & you are bombarded with mafia movies for a long time. You attempt to confuse the system by giving Pretty Woman five stars as well. Then the system tries to gamely proceed by suggesting "Those who watched Godfather AND Pretty Woman are more likely to watch - So I married an Axe murderer."
Can't win.
There are auto-complete screenplay software that basically make a composite of the top 100 best selling screenplays & do what in the industry is called a flip. Namely, change male to female, winner to loser, comedy to tragedy etc. These data-driven screenplay software might suggest that if you take the ladies from Thelma & Louise & replace them with grizzly old men, you get Unforgiven.
There are lyric generation software with the same flavor, umpteen loop generators & infinite jukeboxes, content recommendation systems along the same lines - since you just starred this code sample on angular, you will enjoy this github repo on react,...
Hopefully you see the downside.
Exactly. I don't want what I already have. Give me what I don't have, or better yet something so creative I don't even know I want it yet.
But that's hard, while stupid hill climbers and infinite monkey machines are easy. It will keep working until people tire of it, which judging from the abysmal sales in music is happening. Won't be long before people also stop going to see Batman Redone.
I understand the general argument, and I don't disagree in general. My intuition is that showing 50 inch TVs to someone who just bought one is not ideal [0]. My point is that the specific examples provided (Hollywood blockbusters, etc.) are not accompanied by any evidence or reasoning to convince me that these industries are not doing a good job of satisfying the market.
[0] That said, I've seen lots of counterintuitive but very real phenomena regarding user behavior, so I won't claim to be that confident about this being ineffective. Perhaps people return TVs a lot and buy other ones. I don't have the data.
The data might just show that offering 50 inch TVs to someone who just bought one is actually a rich opportunity. Perhaps both televisions were stolen in a burglary. Or someone is finally upgrading all their televisions from CRT to flat screen -- maybe they moved from a house to a small apartment. It would not surprise me in the least if people were 10 times more likely to purchase a television having just bought one, compared to individuals randomly selected.
And yet you can find plenty of articles about how procedurally generated game environments create longevity.
Imagine a machine that could make procedurally generated movies, would that interest you?
Stories have already been distilled to the Seven Basic Plots [1]
Novelty needs repetion in order to be novel.
[1] http://tvtropes.org/pmwiki/pmwiki.php/Main/TheSevenBasicPlot...
It's seems clear that any content that advances the art is better than a content doesn't advance the art. I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima.
> It's seems clear that any content that advances the art is better than a content doesn't advance the art.
Better in what way?
Let's say Movie A is a well loved blockbuster that millions of people see and enjoy. Movie B is a very mixed piece that isn't really enjoyable to watch, but "advances the art" in some key ways. Movie C is an even more well loved blockbuster than movie A, which even more millions of people will see and enjoy, but that was only made because the director was one of the few that really understood Movie B.
The argument, as I understand it, is that Movie B is somehow objectively better than Movie A and Movie C, because it enables Movie C to exist, even though Movie C isn't actually good, because it doesn't advance the art? That doesn't make sense to me. The journey has value only to the extent that the destination is valuable, no? If C is trash, then what was the point of "advancing the art" enough that we could make C? (Conversely, if we're discussing "advancing the art" in a way that isn't required to make anything anyone wants to watch, then we're clearly not discussing finding a global maxima, right?)
You're retroactively adding new premises (eg that Move C is crap) in order to support your conclusion. Step back from your post for a moment and you'll see it's based on a logical fallacy.
B does not exist for the sake of C, which is profiteering from B.
B exists for D, which will be better than B.
Isn't this the history of progress in creative human activities?
Sounds like an exploitation vs. exploration discussion. The correct answer is that both is worthless without the other; executing movies well is worthless without exploring new movie ideas and vice versa.
Given that we've been exploring for quite some time I guess exploitation is generally preferable if we had to choose.
Executing a movie with new ideas well is not an option as you're relying on the new idea being good i.e. dumb luck.
> It's seems clear that any content that advances the art is better than a content doesn't advance the art.
Perhaps that's true with all else being equal, but clearly all else isn't equal.
> I guess that what he meant is that global maxima is better than local maxima (which is clearly true), and data-driven vision is a hill climbing strategy, thus locking you into a local maxima.
My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum.
It's true in the sense that what makes technology and culture interesting is invention, not endless recycling.
You're pretty much suggesting that using strong feedback to force culture to stay within a tiny area of the total possible cultural phase space is just as interesting as allowing chaotic exploration of the entire space.
It's not just an argument against creativity, it's an argument against invention in general.
>My issue is that none of those examples are backed by any evidence that they are not doing a decent job of finding a global maximum.
That's the thing about global maxima - you only know that you've found a global maximum if you've explored the entire space.
Otherwise you've just stumbled across a local attractor, and you're stuck in a loop around it.
This isn't even a good analogy, because cultural attractors are contingent, and they vary over time. They're also unpredictable.
The reason they capture attention isn't because they're maxima in some analytic sense. They become found maxima because they summarise some aspect of human experience, so they appeal to a lot of people at once.
The spectrum of possible maxima is mysterious and not understood, which is how you get - say - Harry Potter coming out of nowhere and captivating a generation.
Culture is music, not a sine wave. You don't just want a single signal - you want a mix of related-but-different signals running all the time.
> It's true in the sense that what makes technology and culture interesting is invention, not endless recycling.
That may very well be what makes technology and culture interesting to you. It's not necessarily what makes technology and culture interesting to everyone, or even a large portion of people.
> You're pretty much suggesting that using strong feedback to force culture to stay within a tiny area of the total possible cultural phase space is just as interesting as allowing chaotic exploration of the entire space.
I made no remarks even remotely suggesting any of those claims.
> That's the thing about global maxima - you only know that you've found a global maximum if you've explored the entire space.
That may be a useful statement for extremely small problem spaces. It's not useful for the problem space of films. There are a lot of different possible 90 minute long 1080p 24FPS 24-bit color films. Good luck performing a search over that problem space.
>That may very well be what makes technology and culture interesting to you.
Do you really prefer static, stagnating cultures to dynamic and inventive ones?
How do you advance art?
I am not very educated in the field of art. I always find orders of magnitude more beauty in nature than in art.
It's possible to make that exact argument in reverse. Those goals are, if not quite universal human values, products of a rich and extremely broad-based cultural tradition.
In contrast, being data-driven means selecting goals that are easy to measure without any attempt to justify them as intrinsically more valuable than other, more difficult to measure goals.
The reality is that goal-selection is always at least somewhat arbitrary, and words like 'data' and 'science' can be and often are used as a cudgel anyone who might support different (arbitrary) goals.
You can make exceptions to all those rules though. Empire Strikes Back is probably the most obvious example of a sequel that was far and beyond the first and changed cinema.
If you're talking about money-making sequels than Toy Story 2 and 3 were memorable and interesting.
Data is useless for art because art is expected to explore the boundaries of what's acceptable and what's conventional. There is no data for things that haven't even been done yet.
Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'.
It was data that causes so many publishing houses to reject future bestsellers. Sales figures for past children's books told publishers that "Harry Potter" would never work.
We all know how that played out.
As for business, existing data would have never told Jobs to build an iPhone - simply because there really was no data on touchscreen devices.
> Historical data on sales of paintings would've never told Picasso to pursue Cubism, or for Salinger to write 'Catcher in the Rye'.
In that case I wish art would be data-driven, and produce more of the pre-Picasso paintings that are fun to look at and pre-Salinger books that are fun to read.
> ...25 sequels, but none of those films will be memorable.
* Back to the Future 2 * Indiana Jones and the Temple of Doom * Army of Darkness * Goldeneye * Evil Dead 2 * Terminator 2 * Ghostbusters 2 * Captain America: Winter Soldier * Batman: The Dark Knight
Yeah, none of those sequels are memorable. We should stop making sequels and only make original films so we don't waste money on travesties such as these.
Those were not sequels planned for this year.
Feh. What you're saying is only true if 100% of decisions are data driven. Just because Hollywood has chosen to make 25 sequels doesn't mean that Sundance no longer exists. It, and countless events and organizations like it, are celebrated inside and outside of Hollywood.
The implications of the hypothetical extension of this trend are clear to a lot of people. As long as some percent of people are pushing the boundary, culture will be fine. Let everyone else enjoy their sequels and reality shows.
disagree about sequels not advancing the art of filmmaking
although the Star Wars prequels were not great IMO, they certainly brought a ton of technological innovations which have had enduring impact on filmmaking in general
Back in the '90s, I met a guy who made quite a bit of money off specialized construction projects in the developing world. He invested a fair amount of that into a rather popular religious children's series, doing very well in the process. Unfortunately, this led him to assume that he was "in the entertainment business."
He hired a company to research the direct-to-video market and tell him what types of product had the best return per dollar. After a lot of time and money, they came back with a report that said, in essence, that the two top performers in the DTV space were science fiction and vampire horror.
The research was sound. The data were irrefutable. He immediately sank millions of dollars into producing a direct-to-video trilogy about vampires from space.
A few years later, I saw him running one of those hot dog concessions outside a Home Depot.[1]
The commonplace version of this are SMB executives who, primed by HBR articles about data-driven decisionmaking, ascribe magical powers to simple linear regressions, believe that a single model can tell them exactly how to unvaryingly run heterogeneous business lines, think overfit just means you can make the model even better, and try to force-feed explanatory variables down the throats of their stats consultants. Analytics is a powerful and necessary toolset, but it's become harder to keep the c-suite from spiking the sauce.
[1] Hell of a work ethic, though. Not long after that he had a few dozen of those carts operating throughout the region. Never did see him again, but you've got to respect a guy who doesn't understand the meaning of "abject failure."
> A ton of people love all of those things.
This is exactly the problem with completely data-driven decisions.
First, do people really love those things, or given an ever-increasing trend toward those formulaic things, are those the particular formulaic things they have chosen?
Obviously, it's important to use whatever data is available, but isn't the entire notion of a startup to throw away _ALL_ of the prior data and do something that entrenched players won't tend to risk? I think the idea is not to throw away what's fundamentally important to whatever you do because numbers.
BuzzFeed is doing much better at _terrible_ and mostly worthless journalism than they would if they hadn't invented, or at least largely propagated, clickbait, sure. Will they ever, possibly, following this course, reach the value of real journalists? They're a distraction.
Sometimes the data just tells us what the easiest ways to distract people are, but following that principle alone is creating a world with lots of cruft and very little actual value.
Journalism was always a distraction.
>These first three examples are awful examples. A ton of people love all of those things.
So what? A ton of people loved smoking cigarettes, segregation, creationism, and tons of other BS stuff too.
The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments.
Especially since what people "like" is not the objective, "own choice", we make it to be, but can be manipulated by advertising, conditioning, lack (or less coverage) of alternatives and finally lack of education.
The idea about culture is also making value judgements, not just bending over to whatever "people like" as if mere enjoyment is the be all end all of arguments.
But isn't that what the article is about? The song it points as the thing to strive for is Bohemian Rapsody (and not, say, Enter, Evening) exactly because people loved it.
While not Enter, Evening, Bohemian Rapsody has a musical sofistication quite above the examples of statistics and crowd-research driven stuff mentioned above.
People love candy and ice cream, but that's not a good reason to stop eating vegetables. You're overlooking the fact that it's quite easy to condition people's wants, and there's a compelling economic incentive to feed people the cultural equivalent of refined sugar regardless of how it affects their long-term health.
I don't think all this sort of stuff is bad. I'm looking forward to the next Avengers movie, in fact I love a good superhero yarn. At the same time, it's a sad fact that big-budget sequels crowd out investment in small ($10 million or less) indie movies, and there's general downward pressure on budgets at the low end of the market.
The fact that they're successful is selection bias. Of course the examples he give will be successful, because otherwise they would be obscure and readers wouldn't no what he was talking about.
But while he argues that focusing on data is probably not the most effective way of running a startup, the main point of the article is stated in the title: that even if a data-driven startup is successful, it'll have no character.
Obviously not every startup needs character, so take from that what you will.
> Data-driven movies gave us 25 Hollywood sequels planned for this year
You mean, data-driven movie investment. Movie producers don't want the same thing as cinephiles.
True, and given the tiny portion of people who are cinephiles, that seems reasonable.
I am by no means a cinephile, but the Hollywood churn is certainly not what I want to see in movies.
I struggle to call Buzzfeed "data-driven journalism." It's certainly SEO-driven, but I don't think I would call it journalism, first off.
Yeah, not journalism. But I'd call it "data-driven media".
People complain about the tyranny of the mob, and then they obey it. Capitalism.
its funny how all of those shows are the ones that are hated by any one with a bit of intellect.
I think the article goes a bit too far against data. Hits like Bohemian Rhapsody are by their nature freak events and not easily reproducible. Nobody is suggesting (I hope) that you can achieve that kind of success without a healthy dose of luck.
However I agree that for early stage startups data driven decision making can be difficult. My experience is its expensive and you often have very little data.
The other issue is this kinda uncanny valley of false rigor. On the one extreme you have very informal analysis. For example, we tweaked our blog post template and increased newsletter signup rates but I can't tell you exact %s because at this stage we don't track it. We seem to be getting a lot more signups, but perhaps its illusory. That's ok. At our level of traffic it really isn't important. At the other extreme you try to model non-stationary processes and all that and have rigorous control over sources of error. In the middle is where I see many companies with, say, A/B testing, believing they have a high level of statistical rigor but not actually achieving that rigor in practice due to many uncontrolled sources of error. This middle spot, where you have too much faith in faulty reasoning, is where I believe bad data driven decision making resides.
Oh, and on turd polishing: http://www.dorodango.com/create.html
> Hits like Bohemian Rhapsody are by their nature freak events and not easily reproducible.
Just like startups...
Reading the article it reminded me of when the Morotola Razr came out. Before that all the data driven studies had been done, and the ideal handset size and shape had been determined. The Razr went against all the rules (it was far too wide according to these studies) and was a huge success at the time.
The so-called data-driven science have not understand the notion of science. In a minimal sense, science is to produce knowledge. There are two things to it: hypothesis generation; testing the hypothesis. As the history and philosophy of sciences have shown, there is no algorithmic way of generating hypotheses. Or if you generate hypotheses algorithmically, you are still left to figure out whether these hypotheses are ad hoc or not. After all, the history of sciences have given powerful heuristics to reduce the solution space to generate hypotheses to solve or explain problems or facts. Here, whether one picks 'solve' or 'explain' depends on which philosophy of science one picks up.
Whenever I see statistics and data-sciences, I see tons of adhoc bullshit masquerading as sciences/knowledge. It is always easy to come up with a hypothesis to explain a set of chosen facts; in order for that hypothesis to be non ad hoc, it has to predict surprising facts.
As the fad continues, we may hear like robots replacing scientists to produce knowledge about various phenomena. For a best critique of AI, check the book by UCBerkeley philosopher Hubert Dreyfus: what computers can't do, a critique of artificial reason.
“Epistemology”, he whispered.
Using data is good, but “based on” offers a lot of wiggle room. A 10% increase in CTR is nothing to sneeze at, but it does not answer the question of the best use of your engineers’ (or designers’, or marketers’) time. Should they have instead been working on the thing that has a 50% chance of a 20% improvement? How do we account for all the data we didn’t bring to bear?
The data is small, the interpretation is big.
There is also the problem that philosophers call “regress”, which is that every rational decision has to trace back to a premise that one assumes in. Should we be in the business we are in, compared to all the other potential uses of talent? We can’t know that empirically, at root.
+1 for epistemology.
Premature optimization: still the root of all evil.
One of the better takeaways from the article was the notion that being data-driven means you're aiming for average, and you might not even hit it. Aim for the moon, you might only achieve orbit instead.
I watched a CEO make arbitrary layoff decisions based on what the numbers said should be the size of a development organization and the the ratio of developers to QA. The actual software being built was irrelevant to his figures. He used numbers to justify grinding the dev organization into the ground.
I'm not afraid of computers acting like people (AI). I'm very worried about people acting like computers.
Every circuit, every program is based on a principle: comparators (analogue) = NAND (digital) = if statements (software). Machines choose their answer by taking a huge amount of information, and sorting it. By design, this leads to some monstrous conclusions. For example, eugenics might be logically efficient, but it is morally abhorrent.
Taking risks, making mistakes: these are not flaws, they are the very essence of being human.
Test yourself! I guess that everyone on here is very rational (as I am). I only discovered this problem in my character after a conversation with an artist, a good friend from high school. She makes all her decisions based on the heart, rather than the mind. Try to do something totally random! When things make no logical sense, the emotions wake up again. You'll "feel" again. It doesn't matter if that's a good or bad feeling - acting like a machine makes you feel nothing at all. A machine can defend every action it takes, because it's never wrong. But machines can't apologise.
There will be data-driven businesses. They're not actually run by humans (whatever the management says), they're run by machines. Those companies could ultimately be fully automated away. It's far better, as a human, to be creative (even if the most creative thing you can do, like me, is teaching machines how to talk to other machines).
> She makes all her decisions based on the heart, rather than the mind
Maybe I reading the wrong thing into that. I make decisions with my heart and mind.
Heart = I care about my kids and want them to be healthy
Mind = To care for them I vaccinate them and don't use homeopathy
Maybe I'm misinterpreting this but my general experience is someone who "makes all decisions based on the heart, rather than the mind" generally makes some very poor decisions that actually don't lead to the results they want.
@Greggman - I like your Heart/Mind example more approachable variant of Bertrand Russell's "The way forward for humanity is compassion guided by knowledge".
This oversimplifies AI. The human brain is based on simple principles as well. The mechanisms of cells and neurons and neurotransmitters are not hard to understand in isolation. There is no principle difference between an AI that functions on top of millions of silicon circuits and the human brain which functions on top of millions of biological circuits.
While machines are presently lacking in certain capabilities such as empathy and conceptualization, they are still very useful as tools and extensions of the human brain as long as the limitations and pitfalls are understood.
You seem to be oversimplifying intelligence, assuming that it comes directly from neuronal activity. I am not saying its not the case, but I would say that its quite a big assumption given our current level of understanding.
With your example of artificial intelligence, the software running on top of silicon circuits still comes from human intelligence. Where does that ultimately come from?
The most important skill, in order to be data-driven, is to ask the right questions. If you're looking to get to product market fit, the questions you should be asking are very different from the ones you should be asking if you're looking to grow a "good" product. In both cases, data can help you reach your goal, but only if you ask the right questions.
If an early-stage startup tries growth hacking before it reaches product market fit, it will likely end in disaster.
Lots of ignorance in the article and some of the comments regarding data-driven decision making. Does the author realize how much of the civilization around us is built and guided by data-driven decisions? Also, people/companies not being able to effectively use their measurements to their advantage is not really evidence against the idea itself. When your model doesn't work, it's not modeling that's broken, it's just your model.
Gut reactions can take us only so far: they break down as we move away from single human-scale familiar problems (ones that the brain has some built-in, evolved capacity of handling, such as reading other people's facial expressions).
Do you realize how much of civilization was built through intuition?
As great as data is, it is also limited. Why? Because we can never gather enough data. All our data is just a simplification of what's actually going on.
Essentially we're just grabbing data that's generated by black-box tests on systems that are astronomically more complex then we can comprehend. In many cases the data tells only a fraction of the story. It's akin to some alien race trying to understand the a computer desktop by measuring the electrical inputs on a usb port and seeing how that effects the voltage output of the hdmi port.
--
Here's a telling example of the power of intuition by a quote from steve jobs responding to Marc Andreessen inquiring about the "critical" problem the iphone had of "not having a physical keyboard.":
‘They’ll get used to it.’
Any datapoint you gathered on keyboards back in the day would have told you otherwise!
I pretty much agree with you 100%, but I hate it when people use Steve Jobs quotes. He was better at this than almost anybody else, the rest of us need some help and some luck.
>> Everyone tells early stage startups to use data for big strategic decisions.
This is not even remotely true in my experience working at and advising startups. Sure, data is important, but more so for tactical matters like ad performance and A/B testing. Big strategic decisions typically employ far less data relatively, pretty much definitionally since no data exists for "big strategic decisions".
There is a looming disaster of a similar nature coming in healthcare too.
Data shouldn't be used to set goals it should be used to achieve them. It may also tell you when it is not currently possible to achieve your goal. That doesn't mean you should throw up your hands and set goals that the data seems to indicate are achievable because (among other things) that is tantamount to believing we can actually predict the future.
I would love to buy you a beer sometime and discuss that with you. If you have any sources for that epiphany... please let me know.
There is one, and only one reason to be 'data driven'. Or to test one's hypotheses, for that matter. And that is to make sure you aren't fooling yourself; that you haven't fallen prey to the myriad cognitive biases; to prevent your preconceived notions from clouding your judgement of reality, aka what is actually going on.
While data always provides more information, the less strong your prior beliefs, the less informative your experiment will be -- If you believe something and it turns out to be majorly false, you get a nice shift in expectations. If you believe in something and it turns out to be very true, you gain lots of information in terms of quantifying the effect you are looking into.
If you are Google, looking to eck out every last 1/1000th of a penny on ads, yeah, maybe a/b testing the shade of blue of a button can be justified.
The more other companies are "Data Driven" [like the somewhat unfortunate examples the author chose], as opposed to "Hypothesis Driven", the more there is room for somebody else to fry bigger fish.
In other words, it's not the "data's" fault, it is ours.
Oh thank goodness it isn't just me.
This article omits one important aspect: the data isn't used only to create. It's also a guide what to delete/abandon. Sure, it might be influenced by chaotic fluctuations but anyway can help make a decision. Sometimes making any decision is better than wandering around with ambivalent gut feelings.
Personally I enjoyed this article.
I'm working within several different businesses right now, and the consistent theme I'm trying to relate to the folks I work with is that data-driven techniques can take you right up to the edge of what is known to be possible. It's the people who work with the ambiguity there and take leaps into the unknown that ultimately change things. It's fine to want to be part of the pack, but for the really ambitious folks being at the front-edge of the pack is still being part of the pack. Learning to make the move out in front is the hard part.
> Sometimes it works. Sometimes it’s critical. But sometimes it fails, or results in unintended consequences that we may not notice for years.
A bit off-topic, but it explains why we got Windows 8x and the upcoming Windows 10 - data driven metrics.
When will Microsoft learn that developers and advanced users turned off the "phone-home" metrics gathering functions in Windows XP, Vista, 7 and Office?
People want Windows 10 to be Windows 7.5. It would be nice to get some lost Windows XP functionality back and shell bugs fixed that are in since Vista.
Being data-driven is detrimental when it replaces common-sense (talking to users, collecting feedback, improving based on feedback).
Before the internet (and being able to track every single action), successful companies were built. It can be done. Using data to drive decisions has some value, but it's not the end-all solution, it's merely a piece of the puzzle.
Anyone read the Crunchbase profile of the company the author is the CTO for?
"Bipsync provides a research automation platform to maximize the productivity of professional investors. Founded in Silicon Valley in 2012 by experienced investors and software developers at Stanford University, the company uses modern technologies and user-centered design to speed up data capture, automate research maintenance and identify insights that drive better decisions for investors and funds." [1]
I mean, maybe I shouldn't be looking for patterns, because, y'know, data. But it seems oddly conflicting to be pitching a product that encourages the use of data to drive decisions and then publicly condemning... the use of data to drive decisions.
Aside from that contradiction, the company just got seed funding four months ago. It's probably far too early to make decisions about the efficacy of being "data-driven." From personal experience, trying to manage people by telling them, "I'm right, let's do it my way," is terribly demotivating (and very prone to error). Conversely, trying to weigh everyone's input equally and sift out good ideas is an organizational nightmare that creates a ton of complexity. Complexity slows down execution. And who decides on the best ideas?
Creating a mental framework for hypothesis testing and building a product based on optimizing for specific metrics is, in my mind, what being data-driven actually means. There are no inconsistencies or personal biases. It's scalable. You can teach the entire team how to approach the design of a feature as a problem with a testable hypothesis. Politics go out the window as execution strategy is determined by return on investment of engineering resources. Being data-driven doesn't discourage creativity, it just allows you to reframe problems.
Buzzfeed clickbait titles are but a small (and, well, effective) subset of a vast array of largely positive things that come from being "data-driven." Attempting to demonize patterns of logical, rational decision-making because you (personally) don't like one outcome is... well, an anti-pattern. (It happens all of the time. See: The history of the scientific method. ;))
Sure, it's not sexy. But it doesn't need to be. It just needs to work.
In God we trust; all others must bring data.
The problem with the data-driven approach is you might be optimising towards a local maximum.
Only with naive optimization algorithms, of course. There's a whole academic field of study devoted to optimization, and avoiding local maxima is a big issue of course. Hundreds if not thousands of papers are published on it, every year.
I struggle to reconcile the data-driven approach to running products and companies with innovator's dilemma.
It seems like micro-decisions are best made after looking at the data, but macro decisions are not.
Data-driven decisions will get you into local maxima, but you'll get stuck there when there's a good chance that a more radical change would help much, much more.
Vision is a fancier way of saying "dumb luck".
Yes, luck is there, but i think it's only a part of it
This reminds me of Warren Buffet's approach to investing. It's better to be approximately right than precisely wrong. I fear that all we are doing with data is figuring out a precise way to fail instead of focusing on being right.
"But fail often and fast" the cliche goes, but what about the opposite, it should be true by simple negation of logic right?
"Win seldomly and slow", then suddenly collecting data on every useless piece of data becomes futile. You are not focused on winning and without the burden of speed and pressure to screw things up. You are absolutely calm and able to think things through.
Business may proceed as usual: fearful managers might use data to make their jobs more secure, savvy managers might use data to move fast and break things.